Method and apparatus for predicting object properties and events using similarity-based information retrieval and modeling

ABSTRACT

Method and apparatus for predicting properties of a target object comprise application of a search manager for analyzing parameters of a plurality of databases for a plurality of objects, the databases comprising an electrical, electromagnetic, acoustic spectral database (ESD), a micro-body assemblage database (MAD) and a database of image data whereby the databases store data objects containing identifying features, source information and information on site properties and context including time and frequency varying data. The method comprises application of multivariate statistical analysis and principal component analysis in combination with content-based image retrieval for providing two-dimensional attributes of three dimensional objects, for example, via preferential image segmentation using a tree of shapes and to predict further properties of objects by means of k-means clustering and related methods. By way of example, one of a criminal activity and a fraudulent activity event, an intrusion event and a fire event and residual objects may be predicted and located and qualified such that, for example, properties of the residual objects may be qualified, for example, via black body radiation and micro-body databases including charcoal assemblages.

This application is a continuation of U.S. patent application Ser. No.13/753,653 filed Jan. 30, 2013, now allowed, which is divisional of U.S.patent application Ser. No. 12/823,284 filed Jun. 25, 2010 (now U.S.Pat. No. 8,392,418 issued Mar. 5, 2013) which claims priority toprovisional U.S. Application Ser. No. 61/220,248 filed Jun. 25, 2009,the entire disclosures of which are hereby incorporated by referenceinto the present application.

TECHNICAL FIELD

The technical field relates to a method and apparatus for predictingobject properties and events using similarity-based informationretrieval and modeling and, in particular, to the application ofdatabase and modeling software supporting model-based inference ofproperties of object data and events involving the objects recorded in aplurality of databases of information collected from previously analyzedobjects and samples of objects including a prediction of an event ofconsumer fraud or criminal activity.

BACKGROUND AND RELATED ARTS

Database systems and search and retrieval from such databases are known.For example, U.S. Pat. No. 5,911,139 to Jain et al. describes a visualimage database search engine which allows for different schema. A schemais a specific collection of primitives to be processed and acorresponding feature vector is used for similarity scoring. Inparticular, a system and method for content-based search and retrievalof visual objects computes a distance between two feature vectors in acomparison process to generate a similarity score.

U.S. Pat. No. 6,778,995 to Gallivan describes a system and method forefficiently generating cluster groupings in a multi-dimensional conceptspace. A plurality of terms is extracted from documents of a collectionof stored, unstructured documents. A concept space is built over thecollection and terms correlated between documents such that a vector maybe mapped for each correlated term. Referring to FIG. 14 of the '995patent, a cluster is populated with documents having vector differencesfalling within a predetermined variance such that a view may begenerated of overlapping clusters.

U.S. Pat. No. 7,127,372 to Boysworth describes an improvedregression-based qualitative analysis algorithm when a mixture, not in alibrary of spectra, and being an “unknown” is subjected to regressionanalysis of “peaks” in a residual error computed between an estimatedspectrum and a measured spectrum. The process is repeated usinginformation from a retro-regression.

U.S. Pat. No. 7,236,971 to Shatdal et al. describes a method and systemfor deriving data through interpolation in a database system. A paralleldatabase system has plural processing units capable of performinginterpolation of data in parallel.

U.S. Pat. No. 7,318,053 to Cha et al. describes an indexing system andmethod for nearest neighbor searches in high dimensional databases usingvectors representing objects in n-dimensional space and local polarcoordinates for vectors such that a query data vector is requested tofind “k” nearest neighbors to the query vector of the vectors in then-dimensional space.

Haining, “Exploratory Spatial Data Analysis in a Geographic InformationSystem,” The Statistician 47, Part 3, 457-469 (1998) describes asoftware system involving the exploratory spatial data analysis featuresof the ARC/INFO geographical information system and identifies spatialpatterns of geographic locations.

With respect to crime analysis in the twentieth century, two of the mostimportant scientific developments include fingerprint and DNA analysis.Fingerprint analysis has resulted, for example, in the AutomaticFingerprint Identification System or AFIS, and DNA analysis hasresulted, for example, at the federal level, in the development of theCODIS database of DNA profiles which are relevant to crime sceneinvestigation and offender profiling and matching. Over the years, fuzzylogic, artificial intelligence, neural networks, data mining, linkageanalysis, geographic and time of incident graphical analysis and evenBayesian networks have been used to identify patterns in different typesof crimes (arson, burglary, murder, sexual assault among others) and tostudy these crime types by performing linkage analysis among, modusoperandi or MO, crime signatures (e.g., always taking a trophy), ritual,and intentional deception by the offender. There has been a dialog inthe literature between those advocating a Hercule Poirot human inferencemodel versus a more scientific-based model for criminal profiling,especially profiling serial offenders (individuals or groups ofindividuals). David J. Icove et al., in his article “Application ofPattern Recognition in Arson Investigation,” Fire Technology, pp. 35-41,1975 highlights a study conducted with the Prince George's County FireDepartment, in which there is discussed the application of patternrecognition methods to the development of techniques for aiding an arsoninvestigator in determining trends in incendiary and suspicious fires.Dr. Icove further developed an algorithm for crime pattern recognitionat the University of Tennessee and published “Principles of IncendiaryCrime Analysis,” June, 1979, in which Dr. Icove discusses a model calledan arson pattern recognition system (ARPS) for arson-fraud crimes whichcomprises 1) detection 2) prediction and 3) prevention. The modelinvolves the investigation of an intentional incendiary event and addingobject data to a database to continuously improve prevention andprediction of individual arsonists and groups of arsonists. Further workby Dr. Icove is discussed in his publications: “Automated CrimeProfiling,” FBI Law Enforcement Bulletin, pp. 27-30, 1986 and his“Motive-Based Offender Profiles of Arson and Fire-Related Crimes,” FBILaw Enforcement Bulletin, pp. 17-23, 1987. In particular, the 1986automated crime profiling article suggests a system depicted in FIG. 22(PRIOR ART). 1) Violent crime incidents are reported and via 2) mediareports, 3) crime scene processing and 4) VICAP crime reports and 5)violent crime research data are used to build 6) an artificialintelligence system. The artificial intelligence system outputs 7) crimescene profiles and 8) strategic and training implications which are theninput to 1) violent crime incident investigation. Icove claims that 1)useless investigative paths may be eliminated, 2) similar profiles, casestudies and the like may be preserved and recalled, 3) complex criminalnetwork problems may be displayed, 4) decision rules accelerateprocessing time, 5) the expert system will output useful informationbased on the knowledge base, and 6) complex criminal problems may besolved. The arson study included data such as day of week, time of day,full/new moon and so arson crimes of juvenile offenders were identified.Also, in 1986, “Criminal Profiling from Crime Scene Analysis,” Douglaset al., Behavioral Sciences and the Law, vol. 4, no. 4, pp. 401-421,suggests in FIG. 1 thereof a criminal profiling process which involves“Decision Process Models” as element 2 of a complex feedback processsimilar to that of FIG. 22 (PRIOR ART) of Icove, but an emphasis isplaced more on the human criminal profiler than on the use of anintelligent pattern recognition modeling. Anne Davies in her article,“Rapists' Behavior: A Three Aspect Model as a Basis for Analysis and theIdentification of Serial Crime,” Forensic Science International, 55,1992, pp. 173-194, suggests a behavioral model broken into three“aspects” which are modus operandi, sexual and personal gratificationand attitude and intimacy. These, in turn, are broken down intotwenty-three attributes ranging from choice of time and location undermodus operandi to use of “compliments” under attitude and intimacy. Mostrecently, referring to FIG. 23 (PRIOR ART), Craig Bennell et al., in“The Impact of Data Degradation and Sample Size on the Performance ofTwo Similarity Coefficients used in Behavioral Linkage Analysis,”Forensic Science International, 199, 2010, pp. 85-92, (received 17 Jun.2009 and prior art to the extent of the 17 Jun. 2009 version or publicuse earlier than 25 Jun. 2009) shows a more detailed behavioralhierarchy at four levels and comprising thirty-six attributes at itsdeepest level from a blitz attack under control behaviors to stealspersonal items under theft behaviors, the behaviors further includingescape behaviors and sexual behaviors. The Federal Bureau ofInvestigation (FBI) later published: “A Report of Essential Findingsfrom a Study of Serial Arsonists,” Allen D. Sapp et al. (including Dr.Icove), 1994 (113 pages). In this detailed study of criminal arsonactivity patterns, findings include attributes of serial arsonists suchas age, gender, race, physical attributes, marital status, maritalhistory, education, occupation, military record, intelligence quotientand sexual preference. Moreover, the life history of subjects resultedin findings such as criminal history, institutional history, medicalhistory, psychological history including suicide attempts, work history,social and family histories and the like. Motives were characterized asvandalism, revenge, excitement, crime concealment, profit and so on.Numerous tables provide details such as the quantitative result that, of39 cases studied of serial arsonists, 28.2% of these arsonists wereclassified, as “menial laborer” and 23.1% were classified as “skilledlaborer.” U.S. Pat. No. 5,781,704 issued Jul. 14, 1998, to Rossmo,describes “An Expert System Method of Performing Crime Site Analysis.”Rossmo suggests utilizing the Manhattan distance as one means ofascertaining a probable location of a criminal and showing probabilityof the criminal being at the location.

Other systems and database technologies are known which incorporatemultivariate statistical analysis and, in particular, principalcomponent analysis, from patent and non-patent literature and othertechnologies which utilize a geographic information system (GIS).

SUMMARY OF THE PREFERRED EMBODIMENTS

In accordance with an embodiment of a method and apparatus forpredicting object properties using similarity-based informationretrieval and modeling and an aspect thereof, database and modelingtechnologies can infer properties, for example, material composition,manufacturer, recognition of a human, recognition of vegetation,combustion product composition, fire causation and geographicinformation among other object properties of objects collected about theworld from similar previously analyzed objects collected about the worldand having their properties stored in a database. Measurable propertiesof the objects may be stored in one or a plurality of databasesincluding multi-dimensional databases. While exact matches to referencedata may not be expected in response to a query for a similar objectgiven a target object under investigation, an automated search strategymay locate nearest neighbor items, or items within a specifiedneighborhood, with the most similar properties, from a referencecollection and utilize any geographic or other information associatedwith these items to predict properties. Models are then utilized topredict properties of the objects from the similar data. The term“object” is intended to incorporate micro to macro size objects as wellas human and animal objects and remains thereof having three dimensionalshape and properties that may include any of the following: temperature,pressure, texture, shape, color, time or frequency varying data,acoustic, radio frequency spectral data, tissue type, morphology,genetic information, phenotypes, image data and the like, Correlationsmay be with geographic features, such as proximity to a body of water,ancient lakebeds, or particular types of vegetation, identity ofmanufacturer or builder, object identification or signaturecharacteristics, human identification or characteristics and the like,so an estimate is desired of the physical or ethnic source or origin orthe likely characteristics of the source or origin of a target object.

A plurality of databases and a modeling and search capability extendsand exploits already patented similarity-based indexing and searchtechnologies developed at the University of Tennessee. The followingpatents and published applications as well as those identified above inthe Background section are incorporated by reference as to their entirecontents: U.S. Pat. Nos. 7,162,372 and 7,672,790 directed to a method ofresolving DNA mixtures; PCT published patent application WO 2007/0244408related by subject matter to Published U.S. Application No. 2009/0228245directed to DNA peak-fitting, yet to be examined; WO 2008/06719 and U.S.Pat. Nos. 7,624,087; 7,640,223; and 7,664,719 directed to an expertsystem and a related patent application yet to be examined; publishedU.S. Application Serial No. 2008/0040046, directed to associating anunknown biological specimen to a family, yet to be examined; U.S. Pat.Nos. 6,741,983; 7,272,612; 7,454,411 and unexamined related PublishedU.S. Patent Application No.'s 2008/0134195; 2008/0172402; 2008/0109461;and Published U.S. Patent Application No. 2009/0055361 awaitingexamination directed to a parallel data processing system and a methodof indexed storage and retrieval of multidimensional information andorganizing data records into clusters or groups of objects. For example,these applications and patents may describe by way of example, theclustering of fire combustion products and their composition such asthose resulting from a volcano or other natural event or a man-madefire, human beings having a DNA genetic profile categorized intoclusters or groups, machines having a specific manufacturer, plant andanimal life indigenous to a particular region of the world, earth andwater bodies subjected to adverse weather conditions, buildings of acity versus those more associated with a town or village and the like topredict object properties. A database populated with measured propertiesof sample objects, not limited to, but, by way of example, electrical orisotopic measurements, and of trace materials found on or in objects orin environmental samples, such as assemblages of micro-bodies including,for example, charcoal or charred particles along with othermicro-bodies, together with, for example, geographic data related to thesource of the sample objects or environmental samples, as well as theirelectrical and acoustic properties can be indexed and searched usingthese technologies to enable the rapid retrieval of information mostrelevant to an object as broadly defined. The indexing methodologyrelies upon data reduction methods such as principal component analysis,together with data clustering algorithms, to rapidly isolate potentialmatches to predict properties of the object, thus producing a selectionof reference data that best match the measured properties for theobject.

Objects that have been previously collected are analyzed andcharacterized using, for example, electrical, electronic (radiofrequency spectral data), acoustic (audible as well as ultrasound),chemical, mechanical, optical (visible and invisible spectra), andisotopic measurements of components, and other information about anexemplary object. For example, WO2008/153590 and Published U.S. PatentApplication No.'s 2008/0309488 and 2009/0252196 and unexamined U.S.patent application Ser. No. 12/575,320 filed Oct. 7, 2009, to Icove etal. describe a passive microwave detection of a speed, fire or intrusionevent including black body and spectral emission data, also incorporatedby reference in their entirety. Icove et al. have measured passiveelectromagnetic radiation from a fire event of different types, a humanbeing, an airplane and speed of a vehicle where quiet radio frequenciesare suggested for directional, noise-free reception. Such objects, whichmay also include vegetation, provide distinctive data that may correlateto a signature for a target object either alone or in concert with datafrom a micro-body database. No active transmission is required from thesource to the object or need the passive directional antenna emit anyactive radiation at any frequency. On the other hand, the sun provides aconstant radiation source for reception by a black body during daylighthours. Black bodies are known to radiate different levels of radiofrequency across the visible and invisible radio spectrum at differentfrequencies depending, for example, on temperature, pressure and timeand frequency varying characteristics. In particular, for example, theevent of a fire and its residual charred remains emit passive and activereflected radiation that can be measured and compared with known spectradata and the properties of the emitting objects predicted. For example,volcanic ash may be differentiated from smoke particle products of awood fire and those of a chemical fire. Similarly, the event of a firehas been studied as will be discussed further herein and the abundanceof microscopic charcoal particles in micro-body assemblages of fireremains or other residue such as condensed metals or oils correlateswith type of fire and both regional and local fire occurrence.Directional microphones are also known for the collection of sound wavesat sub-audible, audible and ultrasound frequencies. Electrical,electromagnetic and acoustic data provide respective spectral signaturesfor recognition of diverse objects.

Electromagnetic waves are created when charged particles such aselectrons change their speed or direction. These electromagnetic wavesconsist of an electric field and a magnetic field perpendicular to theelectric field. The oscillations of these fields are reflected in thefrequency and wavelength of the electromagnetic wave. The frequency isthe number of waves (or cycles) per second. The energy of these wavesmay also be characterized in terms of the energy of photons, mass-lessparticles of energy traveling at the speed of light that may be emittedat certain discrete energy levels. The following mathematicalrelationship demonstrates a relationship among the wavelength of anelectromagnetic wave, its frequency, and its energy:

$\lambda = {\frac{c}{f} = \frac{hc}{E}}$

where

-   -   λ: wavelength (meters)    -   c=speed of light (3×10⁸ meters per second)    -   f=frequency (Hertz)    -   h=Planck's constant (6.63×10²⁷ ergs per second)    -   E=energy of the electromagnetic wave (ergs)

Wavelength and frequency are the inverse of one another as related bythe speed of light, and may be used interchangeably herein in thedescription of embodiments and the claims as equivalents of one another.Note that the energy of an electromagnetic wave is proportional to thefrequency and is inversely proportional to the wavelength. Therefore,the higher the energy of the electromagnetic wave, the higher thefrequency, and the shorter the wavelength.

The spectrum of electromagnetic waves is generally divided into regionsor spectra, classified as to their wavelength or, inversely, as to theirfrequency. These bands of wavelengths (frequencies) range from short tolong wavelengths (high to low frequency) and generally consist of gammarays, x-rays, ultraviolet, visible light, infrared, microwave, and radiowaves. The term “microwave” generally is used to refer to waves havingfrequencies between 300 Megahertz (MHz) (wavelength=1 m) and 300Gigahertz GHz (wavelength=1 mm). Microwave radiation is highlydirectional, and the higher the frequency, the more directional theemitted radiation. For the purposes of the present application andclaims, an emission above 300 GHz up to 1000 GHz will also be consideredwithin the microwave band.

Radiation via electromagnetic waves can be emitted by thermal andnon-thermal means, depending upon the effect of the temperature of theobject emitting the energy. Non-thermal emission of radiation in generaldoes not depend on the emitting object's temperature. The majority ofthe research into non-thermal emission concerns the acceleration ofcharged particles, most commonly electrons, within magnetic fields, aprocess referred to in the astrophysics field as synchrotron emission.For example, astrophysicists and radio astronomers look for synchrotronemissions from distant stars, supernovas, and molecular clouds.

On the other hand, thermal emission of radiation from electromagneticwaves depends upon the temperature of the object emitting the radiation.Raising the temperature of an object causes atoms and molecules to moveand collide at increasing speeds, thus increasing their accelerations.The acceleration of charged particles emits electromagnetic radiationwhich forms peaks within the wavelength spectrum. There may be a directcorrelation in changes in temperature impacting the accelerations of thecomposite particles of an object with the frequency of the radiation andpeaks within the spectrum. Once an object reaches its equilibriumtemperature, it re-radiates energy at characteristic spectrum peaks.

Similarly, the acoustic spectrum from sub-audible to ultrasound energyat high frequency, for example, 100 megaHertz, may be detected bysimilar directional microphones and their data recorded from objects ina database. Electrical characteristics such as impedance andcharacteristics of an object such as insulation or conduction can beobserved and recorded in a database. A combination database of radiofrequency, acoustic and/or other spectra (for example, optical) emissiondata as discussed herein may be referred to herein as anelectro-acoustic spectral database or ESD.

Common forms of radiation include black body radiation, free-freeemission, and spectral line emission. A black body is a theoreticalobject that completely absorbs all of the radiation falling upon it anddoes not reflect any of the radiation. Thus, any radiation coming from ablack body is from its inherent radiation and is not the result of anyradiation incident upon it. Black body radiation is a basic form ofthermal emission of electromagnetic radiation from an object whosetemperature is above absolute zero (0 Kelvin). Practical examples ofblack body radiators include a human body, a Bunsen burner, a candleflame, the sun, vegetation of different types, water bodies, rockformations, man-made structures, machines and other stars in the galaxy.

Passive high-gain directional microwave antennas and receivers have beenused to measure the temperature of a remote object in the technicalfield commonly known as microwave radiometry. Typical users of microwaveradiometry are radio astronomers scanning extraterrestrial objects andthe earth. A microwave radiometer known from the field of the astronomysciences pointed at the sky can produce a measurable voltage outputwhich is proportional to the temperature of the target. On the otherhand, passive directional radio frequency and acoustic microphones,antennas and receivers pointed toward the earth from an elevatedposition such as a forest fire tower, a building, an aircraft or asatellite may collect spectral data of all types from objects at whichthe directional antennas and microphones are pointed and recognized byhuman observers.

As described above, it is known that fire, including non-flaming firessuch as smoldering embers and volcanic rock, emits a wide spectrum ofelectromagnetic and acoustic radiation. Such radiation includes not onlyinfrared (heat) radiation, but also includes microwave radiation in therange of 300 MHz to 1000 GHz and at corresponding wavelengths of from 1meter to less than 1 mm, due to the energy radiated by such fires asblack body emission and spectral line emission caused by the high energy(temperature) levels of a fire. Such microwave (and acoustic) radiationcan be detected without the need for any corresponding emission ofmicrowave radiation by an antenna. Instead, in accordance with aspectsand features described herein, the emitted spectral energy of a fire andresultant combustion residuals in the microwave regions of theelectromagnetic spectrum and acoustic spectrum can be detected usingpassive microwave and acoustic detection by one or more passivedirectional antennae/microphones.

In addition, living bodies such as persons or animals also emitmicrowave and acoustic radiation due to their inherent thermal energyvia black body emission. This radiation and acoustic radiation also canbe detected by the same directional antennas and microphones used todetect the microwave radiation and acoustic output from a fire. Anelectrical/acoustic spectral database (ESD) of persons, animals,objects, plants, structures, vehicles, machines and the like can beproduced comprising signature spectral (electrical, electromagnetic andacoustic) and black body emission characteristics.

Each measured property can assist in locating or identifying, forexample, the source or predict other properties of the object, forexample, if geographically tagged reference data with similarcharacteristics are available. Trace materials such as charred, forexample, charcoal particles in micro-body assemblages (mixtures withother materials and particles) can be used to identity, if not thegeographic location or origin, then characteristics of that locationsuch as an expected distribution of plant species, soil types,temperature and humidity conditions, and the nearby presence ofgeographic features such as water bodies, ancient lake beds, volcanicrock, other rock formations, man-made structures and machines andoutcrops of sedimentary rocks. The relative abundance of charcoal insamples and the morphologies of charcoal particles in micro-bodyassemblages can provide clues about the prevalence of agricultural,household, or other burning, and potentially of fossil fuel combustionby automobiles or industries. As discussed herein, a micro-bodyassemblage database may be referred to as a MAD database. While a singleproperty of a given object may not provide sufficient discriminatorypower, the fusion of information associated with multiple measuredproperties of multiple objects and micro-body assemblages is more likelyto lead to an accurate geographic or other object property prediction orcharacterization. The above-referenced prior work at the University ofTennessee utilized data obtained from human DNA for clustering. Otherprior work at the University of Tennessee utilized content-based imageretrieval (CBIR) (Z. Shen, Database Similarity Search in Metric Spaces:Limitations and Opportunities, MS Thesis, Electrical Engineering,University of Tennessee, August, 2004) and preferential imagesegmentation of electronic circuit components for clustering (Y. Pan,Image Segmentation using PDE, Variational, Morphological andProbabilistic Methods, PhD Dissertation, Electrical Engineering,University of Tennessee, December, 2007). A resultant image database maybe referred to herein as a content-based image retrieval database(CBIR). Also, the University of Tennessee has reported in non-patentliterature on automated classification of diatoms and the use ofprinciple component analysis methods for identification of environmentalsignatures of micro-body assemblages which include pollen.

Data coding methods for a plurality of multi-dimensional databases thatare compatible with the present similarity-based indexing and searchmethodology support an analysis and exploitation of the correlationsbetween micro-body assemblage data and location/feature and otherproperty prediction data. Databases and related modeling software mayutilize the proposed methodology including, for example, a plurality ofdatabases comprising electrical/electronic/acoustic data (ESD) andmicro-body material assemblage data (MAD) from the literature and CBIRdatabases maintained for objects of interest as will be discussedherein.

Modeling software and related database technology may lead to aninference of the geographic location and characteristics of points oforigin and time/season related data using measurements of objectproperties and associated trace materials and comparison to referenceand historical data. One embodiment comprises a software system thatsupports collection and modeling activities using a variety ofmodalities, including electrical, spectral (electromagnetic andacoustic) and isotopic measurements of samples, and analysis ofmicro-bodies having entrained charcoal particles and other micro-bodiesincluding, for example, diatoms and foraminifera or other micro-bodies,as well as images to identify points of origin and, possibly,time-varying data, for example, transit routes of objects from a pointof origin (for example, associating oil droplets in a body of water orparticulate matter in air with the site of an oil spill or leakage or asource of air pollution). In these applications, objects collected fromfield operations can be analyzed and characterized using, for example,electrical, chemical, acoustic, mechanical and isotopic measurements ofcomponents, and information about trace contaminants. Each measuredproperty can help locate or identify the source of the object or predictother object properties if reference data with known or measuredcharacteristics are available. Trace materials, such as micro-bodyassemblages including charcoal particles and other micro-bodiesincluding pollen, diatoms, and foraminifera, can be used to identify, ifnot a point of origin or transit, then characteristics of that locationsuch as an expected distribution of plant species, soil types,temperature and humidity conditions, and the nearby presence of waterbodies, ancient lake beds, and outcrops of sedimentary rocks. Asexplained above, the relative abundance of charcoal in micro-bodyassemblage samples and the morphologies of charcoal particles canprovide clues about the prevalence of agricultural, household, or otherburning, and potentially of fossil fuel combustion by automobiles orindustries. While a single property may not provide sufficientdiscriminatory power, the fusion of information associated with multiplemeasured properties is more likely to lead to an accurate objectcharacterization and prediction of other object properties that mayfurther include date and time data.

Similarity-based search technologies are incorporated into database andmodeling software embodiments that support model-based inference ofproperties of objects from a database of information gathered frompreviously analyzed objects and samples. An anticipated structure ofthis software is provided in the subsection title “Detailed Discussionof Embodiments.” The software may operate as an overlay to a Commercialoff-the-Shelf (COTS) database product that supports SQL queries across astandard network interface. The MySQL database software from Oracle maybe utilized for this purpose; (refer to http://www.mysql.org/for furtherinformation).

Electrical, electromagnetic and acoustic measurements, specificallytime- and frequency-series data, exist in the published literature forcertain objects such as previous fire events and residual objects.Multivariate statistical analysis, based upon principal componentanalysis (PCA) methods, can be used to extract the data most relevant tolocalization from the raw measurement data. Analysis of spectra usingPCA for identification of chemical compounds and inference of origin hasbeen very successfully employed in the field of analytical and foodchemistry. The extracted content can be utilized to organize a databaseof information about objects in a manner that supports nearest neighborsearch strategies based upon measures of similarities between objects.The methods are highly efficient because of the in-memory databaseindex. The enabling information technologies for this approach aredescribed, for example, in U.S. Pat. Nos. 6,741,983, 7,272,612, and7,454,411 incorporated by reference herein as to their entire contentsfor all purposes. An overview of one of the technologies is providedbelow in the subsection titled “Multivariate Statistical Analysis andData Clustering”. Another method indexes information using partitionsdetermined by entropy and adjacency measurements or functions. Thesepatented methods have been used to construct several different types ofdatabases that implement similarity-based search strategies, includingdatabases of human DNA profiles used for forensic identification andhave also been applied, as will be described below for content-basedimage retrieval (CBIR) databases. As will be discussed herein, by way ofexample, databases of data of different types for a given object maycontain previously collected and stored data of each type in an imagedatabase, a micro-body assemblage database and an acoustic andelectromagnetic spectral database. Other databases having data ofdifferent types may be, by way of example, one of time series andfrequency series data, for example, the maintenance, repair, componentwear and failure record for a machine, vehicle or process, for example,pipe, pump or valve failure or wear or electrical system failure such aspower (battery).

Trace particle assemblages in sediment and soil samples are used byforensic scientists to infer the geographic and environmentalcharacteristics of samples from crime investigations. For example,micro-body assemblages in a soil sample on a shovel, for example,containing charcoal, pollen and the like can provide information onexistent vegetation and vegetation fire residuals that may help pinpointa grave site. This forensic work is discussed, for example, in D. A.Korejwo, J. B. Webb, D. A. Willard, and T. P. Sheehan, “Pollen analysis:An underutilized discipline in the U.S. forensic science community,”presented at the Trace Evidence Symposium sponsored by the NationalInstitute of Justice and held Aug. 13-16, 2007 in Clearwater Beach Fla.Micro-body assemblages including especially charcoal particles or pollenand, for example, foraminifera, and other microfossils can similarlyhelp to establish the origin or travel route of a suspect or objectinvolved in a crime. Such micro-body assemblages are also studied tounderstand past climate and environmental change, and in the case ofpollen, in research on human allergens, crop pollination, and honeyproduction. The use of microfossils in these various applications hasproduced literature on microfossil types and related micro-bodyassemblages that can be used to help develop properties of objects ofinterest. Of particular importance are studies of modern pollen anddiatom distributions carried out to help calibrate records of pastenvironmental change obtained by studying stratigraphic sequences ofmicrofossil assemblages preserved in modern and ancient lake and marinebasins; see, for example, L. M. Kennedy, S. P. Horn, and K. H. Orvis,“Modern pollen spectra from the highlands of the Cordillera Central,Dominican Republic,” Review of Palaeobotany and Palynology 137 (2005)51-68; K. A. Haberyan, S. P. Horn, and B. F. Cumming, “Diatomassemblages from Costa Rican lakes: An initial ecological assessment,”Journal of Paleolimnology 17 (1997) 263-274, and C. Shen, K.-B. Liu, L.Tang, and J. T. Overpeck, “Numerical analysis of modern and fossilpollen data from the Tibetan Plateau,” Annals of the Association ofAmerican Geographers 98 (2008) 755-772. These so-called “moderncalibration studies” have the goal of relating modern micro-bodyassemblage data to patterns of climate, vegetation, and otherenvironmental variables—in which we use the relationships between modernmicro-body assemblages and environmental and geographical factors topredict properties of objects of interest.

Modern micro-body assemblages are usually expressed as percentages ofparticles, for example, charcoal particles, classified by taxonomy or(when taxonomy cannot be resolved) by morphology; see, for example, M.D. Enache and B. F. Cumming, “Tracking recorded fires using charcoalmorphology from the sedimentary sequence of Prosser Lake, BritishColumbia (Canada),” Quaternary Research 65 (2006) 282-292. Themicro-body assemblage data can be treated as vectors and can be readilyprocessed using the similarity-based information retrieval and modelingtechnologies discussed herein.

Source data on modern micro-body assemblages are available in thepublished literature, in unpublished documents such as student theses,and in a few databases developed for paleoclimate research. Informationon these trace contaminants in modern sediments and soils and inpaleolakes (ancient lakes now dry) of the study region, which are asource of airborne micro-bodies (such as volcanic ash) is obtainablefrom published literature and from field study. Although the utility ofthese trace contaminants for source and property attribution has beenestablished, research efforts aimed at formalizing the automated use ofinformation on modern micro-body assemblages and their environmentalcontrols are provided according to one embodiment. Automatedidentification methods for, for example, micro-body assemblagescomprising pollen grains and prior pollen studies in a given region maycomprise portions of a database for pollen, a similar database providedfor charcoal particles and a similar database for each type ofmicro-body assemblage of interest.

Micro-body assemblage data (MAD) collection may comprise anotherdatabase collection and its coding. Geographical locations (coordinates)and environmental characteristics of samples, including climate factors,vegetation, presence of wetlands, soil types, land use, and otherfactors and properties may be included in the database. Again, availableliterature and other data sources potentially include high resolutionaircraft imagery, forest fire tower, traffic monitoring, buildingsecurity and satellite and related spectra and other data collection.Because micro-bodies such as volcanic ash, smoke particles and pollencan be transported long distances by wind, micro-body assemblages may bedetermined, for example, by the matrix of vegetation and naturalstructure types over fairly broad areas. Diatom assemblages may likewisebe influenced by paleolakes upwind from the sampling site. To tiemicro-body samples and objects to geographic and environmentalconditions suggests populating related assemblage databases withinformation on a regional context as well as local characteristics of aparticular sampling site.

A reference core database may preserve both assemblage information andcitations to the sources, and, when it is available, linkage tosupporting imagery for later comparison to target samples usingcontent-based image resolution CBIR. Assembly of a sufficiently largereference database for accurate estimation of geographic characteristicsand location is likely to require a program of environmental sampling tosupplement data available in the literature and world-wide databases.Such a program may include samples from various world regions andeventually require automated sample analysis. An automated particleanalysis system may require computer-aided image analysis andinterpretation, acquired using either optical or electron microscopy orother imaging techniques not limited to X-ray and magnetic resonanceimaging. Some work has been done on the automated classification oftrace micro-body particles using image analysis. Automated systems arelimited to use in a few laboratories engaged in research on automation;see, for example, J. Bollmann et al. “Automated particle analysis:calcareous microfossils,” in Image Analysis, Sediments andPaleoenvironments, P. Francus (ed.), Kluwer (2004) 229-252 and I.France, A. W. G. Duller, and G. A. T. Duller, “Software aspects ofautomated recognition of particles: the example of pollen,” in ImageAnalysis, Sediments and Paleoenvironments, P. Francus (ed.), Kluwer(2004) 253-272. Automated identification approaches typically rely uponartificial neural networks, which require extensive tuning usingtraining sets and do not readily extend to accommodate new information.

European researchers have reported some success for identification ofmicro-body assemblages including diatoms. Identification of micro-bodyassemblages depends upon 3D characteristics that are inferable bymanipulation of particles within a sample to obtain images of individualgrains in various orientations, and rates of correct identification tendto be in the mid-80% range; see, for example, C. Chen et al,“Feasibility study on automated recognition of allergenic pollen: grass,birch and mugwort,” Aerobiologia 22 (2006) 275-284. Microscopytechniques such as refocusing to move the focal plane through the grainand acquiring a sequence of images can be useful, and the issues areless severe with scanning electron microscopy (SEM) and, possibly, withconfocal image acquisition techniques; see, for example, H. Seppä and K.D. Bennett, “Quaternary pollen analysis: recent progress inpalaeoecology and palaeoclimatology,” Progress in Physical Geography 27,4 (2003) 548-579. Partial and overlapping images of particles are alsoprevalent, and obscure features that are needed for identification. Ahuman operator can reorient and separate particles during observation(currently not feasible in automated microscopy systems but likelywithin the near future). An evolving technology for image processing andobject recognition, preferential image segmentation, can be used toisolate features of interest from image data, such as pollen anddiatoms, for use in queries to an image database. This technology isdescribed in Y. Pan, J. D. Birdwell and S. M. Djouadi, “Preferentialimage segmentation using trees of shapes,” IEEE Trans. Image Processing,18 (2009), 854-866, and may be an initial processing step for images ofpollen and diatoms, prior to multivariate statistical analysis andstorage or search in a database. Other known methods of imageenhancement, registration, segmentation and feature extraction areavailable in the published literature and can also be used.

Measured properties of objects and entrained materials can be utilized,in conjunction with a database that supports search and retrieval basedupon similarities among objects, to provide information about points oforigin and time varying data about the object and to predict furtherproperties. A body of information exists in the literature on thegeographic distributions of some micro-bodies including microfossils,particularly pollen grains and diatoms, and on the environmentalcharacteristics of sample collection sites.

Trace analysis of pollen samples in forensic botany relies upon thecorrelations between micro-body assemblages, or the distribution ofpollen grains across plant taxa, and source location and environmentalcharacteristics. The assemblage is simply a vector of percentages ofgrains from each taxon found in a sample and is determined by manualanalysis in a laboratory. Micro-body assemblage data for pollen andother microfossils may be utilized along with raw data such as imagesobtained by optical or electron microscopy. Identification ofmicro-bodies including charcoal particles and microfossils may beautomated using a combination of Content-Based Image Retrieval (CBIR)and a reference database of typed images, with a transition to candidateautomated identification system(s).

CBIR is a relatively new technology that has undergone rapid evolutionover the past decade. An early application is discussed by A. Oakly; seeA. Oakly, “A Database Management System for Vision Applications,”Proceedings of the Conference on British Machine Vision, vol. 2, 629-639(1994), using Manchester Visual Query Language to distinguish twomicrofossils using a projected circular Hough transform in a microfossilimage. The effectiveness of CBIR is dependent upon the range of imagecontent that must be searched. For example, human facial recognitionsystems tend to exhibit reasonably good performance with adequatelighting and standardized profiles and image geometries (for example,the full facial views with flat lighting that are typical of driverslicenses and ID cards), in contrast, a facial recognition system thatuses actively controlled cameras in an outdoor environment to acquiredata from uncontrolled subjects tends to have a poorer performance.

As will be explained herein, CBIR in one embodiment is based on priorwork on preferential, or model-based, image segmentation, and can beused to focus upon those portions of an image (for example, aperturesand sculpturing on pollen grains) most likely to lead to accurateidentification, and the use of similarity-based search strategies todetermine reference objects with similar features. A successful systemmay identify and focus upon micro-bodies including charcoal particlesand microfossils including pollen grains (or diatoms), identify eachgrain, and determine the frequencies of occurrence of each type. Thesedata can then be used in a search for similar micro-body assemblageswithin a micro-body assemblage database (which as described above maycomprise a plurality of databases, one for each micro-body), to providedata relevant to a source or other properties of an object of interestsuch as a smoke particle, man-made fire remnant or volcanic ash.Development of a large-scale trace analysis capability based uponentrained grains in objects, for example, including charcoal, requiresacquisition and coding of additional reference data from the publishedliterature. An automated micro-body assemblage identification system asdescribed herein can substantially reduce the manpower requirements forreference data acquisition and allow better coverage of geographicregions of interest.

Electrical, electromagnetic and acoustic properties of object componentsare expected to be indicative of object properties and may provide anobject signature. A database is known for emission of black bodyradiation from known objects, and this database may be utilized as oneexample of a property of an object. Acoustic or noise emission isanother example of a property exhibiting a spectrum which may be relatedto further properties such as pressure, temperature, object type, suchas a type of human body, machine or vehicle, and vary over time. Thesemeasurements comprise, but are not limited to, spectral data and havebeen shown to correlate to an object such as a human being, a structureor a fire event or its residuals as discussed above. Multivariatestatistical analysis, based upon principal component analysis (PCA) orpartial least-squares (PLS) methods, can be used to extract the datamost relevant to localization from the raw measurement data.

The extracted content can be utilized to organize a database ofinformation about properties of objects and to predict furtherproperties in a manner that supports nearest neighbor search strategiesbased upon measures of similarities between objects. Information aboutsimilar reference objects from the database can then be utilized toestimate or predict properties of an object and the object itself. Newobjects can be treated as new information and incorporated, withappropriate review, into the forensic database, to be used to link toand identify future objects with similar properties. This allows thereference data collection to grow as analyses are performed and maintaincurrency. The database search and retrieval methods are highly efficientbecause of the in-memory database index. The database may includemetadata, such as information about date and time, and source data suchas manufacturer and/or vendor, or location of an object when thisinformation is available. A database search and retrieval operationprovides access to the metadata of objects similar to an unknown targetobject, which provides inferences about the point of origin for each newobject analyzed and searched. By similarity, as used in the application,is intended, by way of example, the probability or likelihood that twoobjects are related by at least one property.

Multivariate statistical analysis presumes that one or more portions ofthe measured characteristics or properties of an object can be expressedas a vector, or ordered sequence, of numbers (of which a large numbermay be required). Values indexed by time (time series) or frequency(spectra) are two examples of such data. A measured concentration orintensity as a function of position, time or another independentvariable, for example, as is used in chromatography or electrophoresis,is another example. While such an ordering may not be appropriate forall measurements of a sample (for example, images, time- orfrequency-series, and genetic sequence data are not always encoded in asingle vector), it is usually possible—and preferable—to represent onetype of measurement as a vector, where several measurement vectors (ofdifferent types) may be associated with each object. Methods such asprincipal component analysis and clustering algorithms (for example,k-means) can be applied to each type of vector, and the methodsdescribed by the above-referenced patents incorporated by reference canbe used to create databases (indexed collections of measurement data)for each vector type.

A single measurement vector, for example, an electrical spectrum, maynot by itself be especially informative of an object's identity,physical and electro-acoustic properties, or location or time varyingactivity. However, the measurement can narrow down the set of possibleorigins or other properties, typically by excluding those referenceobjects that have substantially different spectra, and other measurementtypes can be used to refine the inferred source or property. As anexample, stable isotope ratios, determined using a mass spectrometer,can be used to associate objects with a particular location, and areutilized in forensic science; see, for example, S. Benson, C. Lennard,P. Maynard, and C. Roux, “Forensic applications of isotope ratio massspectrometry—a review,” Forensic Science International 157 (2006) 1-22.Entrained pollen and diatoms can also be used for inference ofgeographic location (or expected characteristics of the location); see,for example, L. A. Milne, V. M. Bryant Jr., and D.C. Mildenhall,“Forensic palynology,” in Forensic Botany: Principles and Applicationsto Criminal Casework. H. M. Coyle (ed.), 217-252. CRC Press, Boca Raton,Fla., 2005 and M. Leira and S. Sabater, “Diatom assemblages distributionin catalan rivers, NE Spain, in relation to chemical and physiographicalfactors,” Water Research 39 (2005) 73-82.

Most chemical elements occur in the environment as a mixture ofisotopes. Stable isotope ratios of Hydrogen, Carbon, Nitrogen, Oxygen,and Sulphur are commonly analyzed in forensic science; see, for example,S. Benson et al., “Forensic Applications of Isotope Ratio MassSpectrometery—a Review,”Forensic Science International 157 (2006) 1-22.Isotope ratios are reported relative to the light isotope as deltavalues relative to a standard, which are the deviation, in percent, fromthe standard. A vector of agreed-upon isotope ratios can be utilized toconstruct an index of stored reference objects and naturally fits withinthe framework of an embodiment of database technologies as describedherein, for example, by creating an index for each isotope ratio.Thresholds can be utilized to exclude reference objects from searchresults if their recorded results are significantly different from thetested sample's values—or accept (meaning one cannot exclude thereference object based upon its isotope ratio), or leave undetermined ifno isotope ratio is available. The results can be combined by returningonly those reference objects that cannot be excluded using at least oneisotope ratio, and that are not excluded using any isotope ratio forfurther analysis.

The use of stable isotope ratios, in addition to the spectral data,points to combining search results across multiple indices. Thisprovides input to the design of an information storage platform: Objectsshould be indexed using multiple and disparate characteristics, such aselectrical spectra and stable isotope ratios, and search results shouldutilize all of the available indexed data which may be all of ESD, MADand CBIR among other data. According to an aspect of an embodiment,first, multivariate statistical analysis and clustering are utilized toextract information that is most relevant to the object from raw datasources, which may assist in determining location or time varyingactivity with respect to an object. Second, search and retrievaloperations are based upon the similarities between objects, and not anexact match to a value in a stored record's field, or inclusion of thatvalue in a specified range. Third, models can be applied to themetadata, or properties, associated with reference objects to predictproperties of interest for a target sample.

The models may be single- or multivariate and may be utilized tointerpolate the value of value set of a property of an object ofinterest for values for the same property of similar objects retrievedfrom the databases. In this case, the property may be, provided by wayof example only, a location or source of manufacture or distribution, atype of material consumed in a fire or used to accelerate or extinguisha fire, the classification of a micro-body or smaller microscopicparticle, the type or class of a vehicle, the type or state of a weaponor other device carried within luggage, or the state or status ofequipment or a process in an industrial setting such as an electricutility or chemical plant. The models may also be statistical, orBayesian, such as a Bayesian network or belief network that relates aset of objects retrieved from the database with an object of interest.This is but one set of exemplary models that are graphs or directedgraphs, as are well known in the field of computer science which canalso be used. In this case, the predicted property may be, for example,the likelihood, probability, or belief that the target object and theretrieved objects satisfy a postulated relationship, or a set oflikelihoods, probabilities, or beliefs determined across alternativehypotheses. If only two hypotheses are postulated, this set oflikelihoods may be expressed as a likelihood ratio. Examples include theidentities, command structure, or purposes of individuals, devices,software components, or other entities such as businesses thatcommunicate via a network, genetic relationships among individuals and,optionally, phenotypes such as the susceptibility to or ability to causeor prevent disease, whether among plants, animals, or single-celledorganisms, and the detection of individuals or other entities engaged inan illicit enterprise. The embodiment further may include imageinformation, which is necessary for identification of pollen, diatoms,and other trace microfossils that may be found on objects including, forexample, vehicles and individuals.

The models may incorporate optimization. One example is the utilizationof optimization such as least squares or maximum likelihood optimizationmethods that are well-known in the art to determine a model that bestfits the values of one or more properties of objects that result from adatabase search. This optimized model can then be used to predict atleast one property of a target object. A more complex example is the useof a database of time series data or data indexed by frequency, such asspectra, obtained from measurements made on a physical process such as achemical reactor or gas turbine. In order to determine or localize aworn or failed component in the process one may record measured data ina database that supports similarity-based or nearest neighbor search atvarious times during the operation of the process. These recorded dataform a historical record of the operation of the process, and recordedmeasurement data from a current operating period can be utilized as atarget in a search of the historical data. Results returned from asearch have similar characteristics to data from the current operatingperiod and can be used to model or predict the status, such as wear orfailure mode, of a component in the process, or to model or predict thefuture behavior of the measured process. For example, similar timeseries data from the historical record can be utilized to develop animpulse response model of the process in order to predict future processstate as a function of time and/or future measurement values. In thiscase, the impulse response model can be obtained by solving a quadraticprogramming optimization or convex optimization problem. Other methodssuch as dynamic matrix control, quadratic dynamic matrix control, modelpredictive control, and optimization of linear matrix inequalities canbe utilized. See, for example, S. P. Boyd et al., “A new CAD method andassociated architectures for linear controllers,” IEEE Transactions onAutomatic Control, 33 (1988) 268-283, C. E. Garcia and A. M. Morshedi,“Quadratic programming solution of dynamic matrix control (QDMC),Chemical Engineering Communications, 46 (1986) 73-87, S. Boyd et al.,Linear Matrix Inlqualities in System and Control Theory, Society forindustrial Mathematics (1997) ISBN 978-0898714852, and M. Morari and J.H. Lee, “Model predictive control: past, present and future,” Computersand Chemical Engineering 23 (1999) 667-682. Approximations to theoptimal solution can also be utilized. See, for example, S. Wei et al.“Applications of numerical optimal control to nonlinear hybrid systems,”Nonlinear Analysis: Hybrid Systems 1 (2007) 264-279, and B. Moerdyk etal. (including inventor J. Douglas Birdwell), “Hybrid optimal controlfor load balancing in a cluster of computer nodes,” Proc. of the 2006IEEE Int. Conf. on Control Applications (2006) 1713-1718. Switchingstrategies may be embedded in a constrained continuous spacerepresenting the fractions of loads to be transferred between each pairof computational elements; see, for example, Bengea et al, “OptimalControl of Switching Systems,” Automatica 41, 11-27 (2005) and Bengea etal., “Optimal and Suboptimal Control of Switching Systems, Proceedingsof the 42^(nd) IEEE Conference on Decision and Control. 5295-5300(2003). A compartmental model can be utilized, where parameteridentification is performed using well-known methods in the art to fitmodel parameters to measurement data; see M. H. Plawecki et al.,“Improved transformation of morphometric measurements for a prioriparameter estimation in a physiologically-based pharmacokinetic model ofethanol,” Biomed Signal Process Control 2 (2007) 97-110. The databasewould be queried to determine the measurement data from the historicalrecord that are most similar to current conditions, determined bymeasurement, such historical measurement data utilized for parameteridentification. In these cases, the predicted or inferredcharacteristics of a target object are utilized to subsequently controla physical process.

Materials exposed to a local environment may entrain small airborneparticles, among which are pollen, charcoal particles, and, for areasnear current or ancient lakes or seas, diatoms and foraminifera. Ofthese particle types, pollen is most commonly used in forensicapplications. Diatoms have been shown to be effective markers for theorigin of objects; see, for example, M. Leira et al., “DiatomAssemblages Distribution in Catalan Rivers, NE Spain, in Relation toChemical and Physiographical Factors,” Water Research 3 (2005) 73-829.If diatoms are aligned flat, i.e. with the polar axis parallel orpossibly at right angles to the image plane, symmetry is helpful, moreso than for pollen grains: see, for example, S. Fischer, “Symmetry BasedIndexing of Diatoms in an Image Database, icpr, vol. 2, 15^(th)International Conference on Pattern Recognition, (2000), 895-898. Seealso E. Joynt et al., “An Image Database for Diatom Identification andNomenclature—Focus on Metadata,” Journal of Paleolimnology 22, 109-114(1999) for a discussion of a relational database for communication oftaxanomic and, ecological information including an image table, anomenclature table (DIATCODE), a collections table, a count table and amethods table. A common use of diatoms in forensics is determiningcircumstances of drownfing. Micro-body assemblages including charcoalparticle and other micro-bodies such as pollen in the assemblages areless often considered in forensic applications. However, studies ofmodern microfossil distributions carried out by paleoenvironentalscientists to calibrate records of long-term change have shown that allof these particle types, charcoal, pollen, diatom, etc. showgeographical patterns and environmental relationships that make thempotentially useful for sourcing objects in forensic applications.Airborne dust in arid regions may contain diatoms and foraminifera fromthe surface sediments of dry lakes that can potentially help to sourcethe dust, given databases being available on the micro-body assemblagesin dry lakes of the region. Marine diatoms may also be found in dust ofarid regions. In dry environments, diatoms, foraminifera, and charcoalparticles may be more resistant than pollen grains to destructionthrough oxidation, giving them special importance for sourcing objectsin some regions of the world.

For all four types of particles, the characterization of a sample is bythe assemblage of micro-bodies including microfossils present, which canhelp to “fingerprint” a source region and thus predict other properties.For pollen, L. A. Milne et al., “Forensic Palynology,” Forensic Botony:Principles and Applications to Criminal Casework 2005, 217-252, CRCPress, recommend identification of 200 to 1000 pollen grains to obtainan adequate characterization. Identification is based upon the physicalcharacteristics of pollen grains, including grain size, shape, andpositioning of apertures (openings in the surface through which thegenetic material is transferred), sculpturing (ornamentation or patternson the surface), wall composition and preservation (which depends uponthe environment at the collection site), and wall structure. Thephysical characteristics are observed after chemical and physicalprocessing to remove other materials, and can utilize either optical orelectron microscopy. The micro-body assemblage data is characterized bythe frequency of occurrence of each type of particle or grain in thesample and can therefore be represented as a vector. If the vector isconsidered to represent the frequency of occurrence of each grainspecies represented in a reference database, it is sparse, since arelatively small number of species (compared to the content of thedatabase) are likely to be present in each sample. In some cases, thepresence of a unique pollen type in a sample may provide a locationalfingerprint, if that particular pollen is produced by a plant specieswith a very narrow distribution. Similarly, it is known that passivedirectional spectral data collection may distinguish one form ofvegetation from another in an ESD database.

At present, grain identification relies upon features visible through amicroscope or in images of grains and human expertise to correlate thosefeatures with grain type and taxonomy. Some digital repositories oridentification keys exist, for example the Newcastle Pollen Collectionand the key to pollen of the Bahamas (www.pollen.mtu.edu). The qualityof images, and the extent to which these resources can be used foridentification, also varies. Most pollen analysts use these digitalresources, or printed volumes of pollen illustrations that predatedigital efforts, only as guides to identification, supplemented byphysical collections of reference pollen slides that analysts develop bysampling herbarium specimens of plants of their study areas, andprocessing them chemically to prepare slides that are similar to slidesprepared from environmental and forensic samples. However, as imagequality in digital collections improve, and as microscopy itself comesto rely more on image analysis, image databases will become more widelyused in pollen identification as well as in the identification of othermicrofossils. In the European community for automatic identification andclassification of diatoms, a project known as ADIAC reportsidentification rates of 80-90%.

A micro-body assemblage may further comprise a heavy metal. A.Facehinelli et al. have studied the regional variability in northernItaly of heavy metals in soil samples, for example, chromium, cobalt,nickel, copper, lead and zinc using MVS and GIS-base approaches; see A.Facchinelli et al, “Multivariate Statistical and GIS-based Approach toIdentify Heavy Metal Sources in Soils,” Environmental Pollution, 114,313-324 (2001). Measures of chemicals and contaminants in dust andsurface soil may provide a fingerprint to link samples to types ofplaces to an actual location such as northern Italy or to other samples.

The design of a grain database may employ Content-Based Image Retrieval(CBIR) using measures of similarity between segments of images. Thesesegments can be grains, or features on the surface of grains (sculptureand apertures). One advantage of retrieval based upon similaritymeasures is the potential to correctly identify degraded grains, orgrains from images that are partially obscured by other grains orartifacts. Prior work includes the extension of CBIR to preferentialimage segmentation and identification using content models based upontrees of shapes; see, for example, Z. Shen, Database Similarity Searchin Metric Spaces: Limitations and Opportunities, MS Thesis, ElectricalEngineering, University of Tennessee, August, 2004 and Y. Pan, ImageSegmentation using PDE, Variational, Morphological and ProbabilisticMethods, PhD Dissertation, Electrical Engineering, University ofTennessee, December, 2007. Metadata associated with stored images ofgrains can include the location and date of collection, as well as otherdescriptive data. Similar database and identification andcharacterization approaches can be utilized for pollen, diatoms, andforaminifera. Studies of microscopic charcoal particles as indicators offire regimes have focused on total particle abundance per volume orweight of sediment, or in comparison to pollen abundance based on visualquantification of particles on slides prepared for pollen analysis; see,for example, K. Anchukaitis and S. P. Horn, “A 2000-year reconstructionof forest disturbance from southern Pacific Costa Rica. Palaeogeography,Palaeoclimatology, Palaeoecology 221 (2005)35-54 and L. M. Kennedy, S.P. Horn, and K. H. Orvis, “A 4000-yr record of fire and forest historyfrom Valle de Bao, Cordillera Central, Dominican Republic,”Palaeogeography, Palaeoecology, Palacoclimatology 231 (1996) 279-290.The feasibility of automatic quantification has been demonstrated; see,also, L. M. Kennedy, S. P. Horn, and K. H. Orvis, “A 4000-yr record offire and forest history from Valle de Bao, Cordillera Central, DominicanRepublic,” Palaeogeography, Palaeoecology, Palaeoclimatology 231 (1996)279-290. Newer approaches have focused on particular charcoalmorphologies that can indicate the type of material burned, which mayprovided more detailed environmental clues. Both approaches to charcoalquantification may be accommodated in an embodiment of the presentdatabase.

The application of the present embodiment is not limited to firedetection, forensics, fire residual determination and the like. Otherapplications include financial, data mining, criminal activity patterndetection and disease modeling or disease discovery.

For example, with respect to a financial application, time series can bestock or other equity prices, and the correlations between time seriescan be used as a measure of similarity (for example, R²) in statistics.One would look for exploitable patterns—equities that move as a group,or that may have correlation delayed in time with respect to another.PCA can be used to cluster similar time series, corresponding toequities that behave similarly. The model can be a method of portfolioanalysis—in other words, an optimal allocation strategy to determine thebest allocation of investments. See also data mining, below.

With respect to data mining, the method can be used to mine informationin a database—looking for clusters of similar behaviors. This can bepurchasing patterns of consumers, or of businesses (e.g., rawmaterials). A model can be applied to some or all of the members of acluster (similar objects) to determine their relationship. The model canbe a Bayesian or belief network, or a pedigree, which is a graph ofhypothetical relationships between objects. Relationships can be flowsof capital or goods/services between members of a cluster (or a subsetof a cluster). Multiple hypothesis testing or maximum likelihoodestimation can be used to determine which relationships (models) aremore (or which is most) likely. Similarity-based search can determineobjects in a database that are most similar to a target, or objects mostsimilar to each other. By exploiting the high speed of the database, onecan perform a search of the database against itself to determine a listof the most similar clusters or subsets of objects and apply models tothese to test hypotheses. The results of this procedure can be used toadd information to the database, which could be “metadata”, or dataabout the data (clusters), mining the database for knowledge.

With respect to detection of patterns in criminal activity, behaviors(objects in the database) may be suspicious transactions that areobserved or reported. Hypotheses may be the possible organizationalstructures of a criminal network or conspiracy. Similarity basedsearches may be used to detect patterns of criminal or fraudulentactivity. This could also be interactions among computers orcommunications devices such as nodes in a communications network, wherethe goal is detection of organized behaviors. Transactions could also bemedical records or medical data such as medical claims for reimbursementfrom insurance or Medicare, where the goal is detection of patterns ofactivity indicative of consumer fraud.

With respect to disease modeling or drug discovery, attributes can bemeasurable quantities about objects, such as individuals, and propertiesthat are inferred by the models and can be an expression ofcharacteristics of the objects, such as disease or drug resistance. Thisrelates to the classic application of Elston and Stewart (R. C, Elstonand J. Stewart, A General Model or the Genetic Analysis of PedigreeData, Human Heredity 21 (1971), 523-542) and models derived from theirapproach with genotypes and phenotypes.

BRIEF DESCRIPTION OF THE DRAWINGS

One embodiment of a method and apparatus for predicting objectproperties will be discussed in the context of the following drawingswherein:

FIG. 1 is a Venn diagram showing selected reference data from threesimilarity searches and their juncture or overlapping region.

FIG. 2 is a graphical overview of the architectural components of aclient-server database supporting similarity-based indexing, search andretrieval according to one embodiment using multiple search engines.

FIG. 3 provides sample “s” attribute vectors associated with storedobjects and a target object denoted by “x” defined by first and secondprincipal component vectors forming a cluster.

FIG. 4 illustrates a modeled behavior for the cluster of FIG. 3 usingdata from selected reference objects (o, *) to estimate the property'svalue for target object x of FIG. 3.

FIG. 5 provides sample images from a database of over one thousandimages for rowing competitions.

FIG. 6 provides images retrieved by a CBIR process of one embodiment fora racing shell.

FIG. 7 shows a projection of image attributes onto two principalcomponent axes, showing data which satisfy the search criteria withinthe central area and marked as such.

FIG. 8 shows a scree plot of cumulative contributions made by aplurality of principal components.

FIG. 9 shows Mahalanobis scores for depicted principal components of10,000 data profiles for alleles in d13s317 and d16s539 in nineclusters.

FIG. 10 shows the make-up of the second principal component for thed13s-d16s data set.

FIG. 11 shows the make-up of the third principal component for thed13s-d16s data set.

FIG. 12 provides a graphical depiction of a database index constructedfrom the results of multivariate statistical analysis and a rankingstrategy.

FIG. 13 provides a histogram of times required to search a 100,000 DNAprofile database for an exact match to a profile.

FIG. 14 illustrates automatic recognition of pollen grains usingsimilarity search.

FIG. 15 shows partition of a 2-level indexing tree wherein FIG. 15Ashows a partition at level 1; FIG. 15B shows a partition at level 2; andFIG. 15B shows a final partition.

FIG. 16 shows a triangle inequality.

FIG. 17 shows a search method using reference points wherein FIG. 17Ashows a reference points P₂ and P₃; FIG. 17B shows reference points P₁and P₂; and FIG. 17C shows reference points P₁ and P₂ from FIGS. 17A and17B.

FIG. 18 shows a performance comparison of two data extraction methods.

FIG. 19 shows preferential image segmentation results wherein FIG. 19Ashows a truck image from FIG. 19C and FIG. 19B shows an electroniccomponent taken from FIG. 19D.

FIG. 20 shows a block diagram of a data modeler platform.

FIG. 21 shows a block diagram of exemplary system components.

FIG. 22 is a prior art block diagram of an automated crime profilingsystem.

FIG. 23 is a prior art behavioral hierarchy at four levels andcomprising thirty-six attributes for a sexual assault crime.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Embodiments of a method and apparatus for predicting object propertiesusing similarity-based information retrieval and modeling will bedescribed with reference to FIGS. 1-21. One embodiment promotes the useof a federation of database indices, for example, those corresponding toelectrical spectra, isotopic ratios, pollen, charcoal, diatoms, andforaminifera, all of which previously stored data can be searched usingsimilarity-based methods for reference samples with characteristicssimilar to those characteristics or of a type of data measured andstored for an object. Each database index may be implemented by a searchengine, utilizing a common commercial off-the-shelf database or a filesystem as a persistent storage repository. A search management processmay be utilized to coordinate requests from clients for informationacross all database indices, and for interpretation of the searchresults to present a coordinated response to each user. Referring toFIG. 1, there is illustrated a Venn diagram of the selected referencedata from three similarity searches. The combination and utilization ofinformation from searches involving multiple attributes of the newlyacquired target object of interest can refine the estimate of propertiesrelated to the object's geographic origin or other property, asillustrated by the central intersection of the three depicted similaritysearches shown as ellipses in FIG. 1. Similarity measures can be used tocluster by similarity and then apply model(s) to clusters to testhypothetical relationships—with or without a target object or object ofinterest. The object of interest may be a member of the database. Forexample, one may perform searches for similar objects contained in thedatabase for all members of the database. See, for example, the thirdmethod of U.S. Published Patent Application No. US 2008-0040046 of Feb.11, 2008, for a method to detect and identify relationships among familymembers in a mass disaster setting. Of course, the embodiment of FIG. 1is merely illustrative and the figure is not limited to three but mayinvolve hundreds or thousands of similarity searches. A database can bedynamic, with new information being added to the database, which datacan change the membership of clusters and the validity of hypotheses.

Reference data that are tagged with properties such as the circumstancesof manufacture or distribution and points of origin can be utilized toinfer point of origin information for a newly acquired target object.Deterministic or static models may be utilized to infer these propertiesand predict other properties. Example models include methods known inthe art of curve fitting and interpolation, least-squares models, andprincipal component analysis (PCA). Maximum likelihood estimationmethods (e.g., likelihood ratios) may also be utilized to provide aquantitative statistical assessment of the relative support forcompeting hypotheses. Likelihood ratio analysis may require investmentin the development of a reference data set because such methods areBayesian—they rely upon a priori statistical information about theprobability that samples with specific characteristics will be found atthe stated locations (or with these characteristics). Other approachessuch as artificial neural networks, belief networks and fuzzy logic maybe used to advantage. Dynamic machine learning methods such asreinforcement learning can be used to update behaviors of the modelsbased upon newly acquired knowledge.

Existing technologies that enable the development of database andmodeling capabilities to support source identification from tracemicrofossil and electrical analyses are utilized in existing systems forthe dynamic indexing and storage of updated data, for example, relatedto human identification. The technologies are utilized in one embodimentto implement a more general type of database (the Core Database) thatsupports utilization of correlations between observed attributes andproperties of reference objects to model and predict the site propertiesand contexts of newly acquired objects of interest.

One embodiment of a database is an Electrical/Acoustic Spectra Database(ESD) that supports the comparison of objects of interest to a referencecollection based upon measured spectral characteristics and inference ofproperties and contexts of one or more target objects from dataassociated with similar reference objects. As described above, such adatabase may comprise black body and spectral emission at acoustic andelectromagnetic radio frequencies (where “acoustic” may compriseaudible, sub-audible and ultrasound spectral emission of objects. Datamay be collected from reference objects at all frequencies and over timeand at varying temperature and atmospheric pressure.

A second embodiment of a database is a Micro-Object Assemblages Database(MAD). MAD supports comparison of micro-object data derived from expertstudy of objects of interest to stored reference micro-body assemblages,and inference of properties and contexts of one or more target objectsfrom physical, geographic and/or environmental data associated withsimilar reference assemblages. Both databases are merely exemplary ofother databases that may be employed to advantage such as acontent-based image retrieval database (CBIR) database as will bedescribed further herein. The databases will support storage of dataobjects containing identifying features (spectra for the ESD,assemblages for the MAD and images for CBIR), source information (suchas when/where/from what a specimen was collected), and information onsite properties and context that can be used to infer geographiclocation and/or time-based activity. Multiple ESD, MAD and CBIRdatabases may be implemented using the Core Database technology to forma plurality of hierarchical and relational databases to improve theaccuracy of the inferred properties of target objects and theirprobability of occurrence.

For example, domestic farming activities may benefit from an ESDdatabase as described above. Since the technique is also sensitive tobody temperatures within the field of view of the receiving antennae,the tracking and corralling of livestock such as cattle over ranges,entering corrals, and even wandering outside boundaries could bebeneficial, particularly for those in the milking industry. Thistechnology could also determine thermal signatures of livestock, humans,or predators so that such animals can be monitored and undesiredintruders identified. Thermal signatures of plants and vegetation maysignal a problem of oncoming drought or plant disease.

Microwave speed, fire and intrusion detection capabilities can also beused to detect the movement of vehicles along roads and tunnels andshipboard and airplane movements along channels. Signature analysiscould identify the traffic flow and thermal signatures differentiatingbetween cars, trucks, motorcycles, planes, trains, boats and othervehicles or vessels. This technique could also identify stalled vehiclesor those catching fire, particularly in high density undergroundtunnels.

The ESD and the MAD and other related databases may have a client-serverarchitecture, as described in the subsection titled “Design Overview”,so both client and server software may be utilized. An example ofinformation on site properties and context is the geographic location,date, and time of collection. However, the information may bedescriptive, especially when reference materials are extracted from theliterature; examples include local and regional vegetation, and theproximity to paleolakes. This information may exist in the primaryliterature, but it also may have been obtained from other sources. Datacoding can be designed to provide information in a form that can beutilized to infer the characteristics of the source of a newly acquiredsample. It is anticipated that a significant portion of the client andserver software will be common to both (or several) databaseapplications. The initial databases and related software provide a baseplatform for other database applications in the future, includingsupport for micro-body including charcoal particle and microfossilrecognition. The database server and associated data processing methodsmay be implemented, for example, using the C++ or a similar programminglanguage, and a client device may be implemented using Java, C# or otherprogramming language suitable for implementation of a user interface orclient program.

Tagging in the ESD and MAD databases may uniquely identify the objectsand selected properties. Multivariate statistical analysis andclustering will play a key role in identifying nearest neighbors andclustering. Matlab may be utilized to provide a rapid prototypingcapability to assess design and data analysis alternatives. Clusteringopportunities may determine an efficient indexing and search method tobe used for the database. One approach is illustrated below, by way ofexample, using DNA profile data in the subsection titled “MultivariateStatistical Analysis and Clustering” (MVS). Electrical and acousticspectral data are, at a fundamental level, vectors that can be processedand aggregated using methods based upon principal component analysis(PCA) and clustering algorithms.

The indexing method may be entropy/adjacency, and is not limited to MVSor PCA. These methods may be used in combination. Entropy measures theability of a node in a database index to segregate data in a collection(subset of the database) into two or more portions of roughly the samesize or complexity. Adjacency measures the ability of a node in adatabase index to impose structure on these portions that preservesimilarity—meaning that similar objects are in portions that are similar(a hierarchical data model where if you want to search for an objectnear (or contained in) portion A, and if the neighborhood of results ofinterest is sufficiently large, you also want to search for objects inportion B (or multiple portions) where the data in portion B is moresimilar to the data in portion A than other data in the database. Thereis a trade-off between entropy and adjacency—our prior work found that asubstantial gain in adjacency can be obtained at the expense of a smalldecrease in entropy (or increase, depending upon the sign that isused—either information gained from applying the query, or entropy ofthe resulting portions).

Examples of indexing methods include: (a) indexing of sequences,including text (words) or characters, using a measure of edit distance,which, when properly defined is a metric (and therefore the metric spaceindexing methods described in Z. Shen's MS thesis work). (b) indexing ofsequences of numbers using a measure of the correlation between thesequences, such as R² or Mahalanobis distance, or inner product ofvectors. (c) A similarity between labeled fragments (such as STR DNA)can be defined as described in our database patent family. (d) indexingcan be based upon similar hierarchical decompositions of objects, suchas the tree of shapes and shape descriptions of segments in images, asused by Y. Pan in his PhD dissertation and the IEEE Trans. ImageProcessing paper, and (e) 3-d structures such as organic compounds andnanoscale structures can be indexed based upon their structuralsimilarities, using, for example, a spanning tree of an annotated graphrepresenting the structure, with term rewriting rules to determinesimilarities in structure (creating, in some applications, anequivalence relation on the set of possible spanning trees and a measureof similarity between equivalence classes). This can also be used todefine the similarities in the structural descriptions of microscopicparticles such as charcoal, pollen, and forams. (f) Finally, indexingcan be based upon metric space methods by embedding objects in a metricspace (or associating objects with elements of a metric space) and usingan inverse of the metric, such as an additive or multiplicative inverse,evaluated upon a pair of objects, as a measure of the objects'similarity.

As micro-body assemblage data are collected during a survey, they may bearchived in a form that can be utilized to populate an operational MADdatabase. Micro-body assemblage data and their associated informationmay be coded for insertion in the MAD database. There exist alternatemethods of data coding for information related to assemblages todetermine a coding framework that best suits the needs of the end usercommunity and supports the requirements of the extant similarity-basedindexing and search technologies.

There are significant sources of micro-object, for example, charcoalparticle and microfossil, assemblage data in the literature andelsewhere, and data can be gathered, for example, for property andcharacteristic dimensions. MAD may store associated environmental andgeographic information for source identification or attributeapplication. Additional data collection and analysis efforts can augmentthe data available in the literature. Automation of portions of thetrace particle analysis process may be achieved utilizing a combinationof preferential image segmentation and a database of typed referenceimages that supports content-based image retrieval (CBIR). One or moredatabases may be developed for Trace Particle Image Recognition (TPIR).For example, charcoal particle and microfossil image data can becollected as part of a literature survey and utilized to evaluateconcepts and databases that are soon available or under development.

Design Overview

This section provides an overview of the design of a database thatimplements efficient similarity-based, or nearest-neighbor search. Thismeans that a request to search the content of the database will returnidentifiers for objects that are within a specified distance to areference, or target, object but may not precisely match the target'scharacteristics. One way to define the term “distance” uses a metricthat is defined on the stored objects, and that can quantify thedissimilarity between two stored objects. A metric satisfies thetriangle inequality, and this fact can be exploited in the design of adatabase index. See, for example, the previously cited MS thesis by Z.Shen and the section below titled “Content-Based Image Recognition”.However, a measure of distance does not have to be a metric. Forexample, see U.S. Pat. Nos. 6,741,983; 7,272,612; and 7,454,411 for moregeneral indexing structures that rely upon concepts of “distance” thatare not all metrics.

Several metrics may be defined and utilized to satisfy a request tosearch the database, in which case the returned identifiers refer toobjects that are within a specified distance to the target object withrespect to each metric. There are performance advantages that can beachieved when the stored objects can be represented as values in avector space and/or when a metric can be used as a measure of distance,or to define the similarity of objects, but this is not necessary and isnot feasible in all applications. For example, images may be representedas values in a metric space that is not a vector space, and DNAsequences and profiles require a looser interpretation of the term“distance” (using mappings that do not satisfy the triangle inequality).Even in these applications, high performance databases have beenimplemented using the methods developed at the University of Tennesseeas described in the issued patents. To enhance readability, terms thatrefer to components and concepts that have particular meaning in thedesign are printed in italics.

FIG. 2 provides a graphical overview of the primary architecturalcomponents of a client-server database supporting similarity-basedindexing, search and retrieval using multiple search engines. Thedatabase (or preferably a collection of databases) utilizes aclient-server architecture that provides simultaneous services tomultiple clients. Architectures have been implemented that leverage theadvantages of parallel computation, using both clusters of computernodes and single nodes with multiple processors and cores. A commercialoff-the-shelf (COTS) database 200 or a computer or network file system(referred to herein as a “COTS Database”) can be utilized for persistentstorage, while the high-performance in-memory indexing and searchtechnologies are implemented in Search Engines 210(1) to 210(n) thatoperate as cooperating threads or tasks within the overall architecture.A Search Manager 220 provides coordination between the Clients 230(1) to230(m), a COTS Database 200, and Search Engines 210(1) to 210(n), aswell as the initial connection protocol for the Clients 230(1) to230(m). The application can be parallelized by allocating separatecomputational resources to each component, such as a Search Engine210(1) to 210(n), by allocating multiple computational resources to anycomponent, as occurs in a Search Engine 210 that utilizes multiplethreads, or using a combination of these methods. Communications amongcomponents in a parallel implementation may be effected using acommunications medium such as a computer network or using shared memory.

A simple example illustrates the design concept. Suppose a databasecontains 14 objects, and that each object is described by a vector ofattributes that are real-valued. Preprocessing of data can be by dataextraction or filtering, such as low or high pass filtering, or Kalmanfiltering or extended Kalman filtering (both using a model ofrelationships among members) or parameter identification. Theseattributes can be analyzed using multivariate statistical analysis(MVS), for example, using principal component analysis (PCA) asdescribed in a subsequent section, to determine a smaller dimensional(two in this example) subspace of the attribute space in which theobjects can be clustered into groups (three in this example). In thissimple example, assume that a measure of similarity between objects,using the projections of the attribute vectors onto the principalcomponent basis vectors for the subspace, is the inverse of Euclideandistance between points. This situation is illustrated in FIG. 3 showingprojections of sample ‘s’ attribute vectors associated with storedobjects (o, *) and an unknown target object (x) onto the subspacedefined by the first and second principal component vectors. Theprojection shows three clusters of objects, of which one containsobjects most similar to the projection of the target object's attributevector, a cluster of five “*” points at the right of FIG. 3 with thetarget object's vector x in the center. In FIG. 3, the rightmost clusterin the figure (containing data points corresponding to five objects) isthe cluster of greatest interest.

The right-most cluster in FIG. 3 illustrates the effect of a similaritysearch operation on the database. The point depicted by the ‘x’ symbolin FIG. 3 corresponds to a newly acquired target object, and the searchoperation identified five stored objects that are most similar to thetarget using the similarity measure defined by the combination of aprojection operation onto the subspace and the Euclidean distancemetric. Suppose the objective is to estimate a property of the targetobject from the property's values for similar objects. This isillustrated in FIG. 4, where the vertical axis represents the value of asingle real-valued property. An algorithm is applied to the similarobjects' data, as represented by the gridded surface, to provide aninterpolated or inferred estimate of the property for the target,represented by the vertical height of the ‘x’ symbol in FIG. 4 (towardthe center of the curved planar space).

A feature of the technical solution of the embodiment is the ability torapidly select objects from a large database that have similarattributes to those of a target object of interest, even when thedatabase contains millions to hundreds of millions of objects. Thisenables the rapid modeling and prediction of a target object'sproperties using data from the set of similar objects.

Referring to FIGS. 5-7, a slightly more complex example furtherillustrates this process. A database was assembled of roughly 1,000images taken by Birdwell at rowing competitions, along with attributesextracted from each image. The similarity-based database index wasconstructed using the attribute value set and utilized to rapidlyidentify objects (images) within a specified metric distance from atarget image. FIG. 5 shows four sample images from the database,illustrating the variety of image data. The top left image in thisfigure was utilized as a target object in a similarity-based search andretrieval operation, yielding images satisfying a specific (fairlystringent) match criterion. The images retrieved using this search areshown in FIG. 6.

FIG. 7 shows a plot of projections onto two principal component axes ofthe attributes of all images stored in the database, with each imageshown as a dot or space. In this example, the retrieval operation was tofind images that had attribute values within a specified 1₁ norm of theattribute value of a target image, where the attribute is vector-valued.The database index allowed the search operation to be narrowed down tothe set of dots shown in a box, where the darker outlier * (RED) dotsrepresent images that satisfy the final similarity constraint. Onemeasure of the efficiency of the database search operation is therelative size of the set of outlier * (RED) dots compared to the set of(green (within the square)) colored dots, since red dots outside thesquare represent images that had to be visited by the search engine torule out their inclusion in the result set (within the box). To continuethe analogy with modeling, the images that correspond to the green dots(in the square) shown in FIG. 7 would be used in a predictive model toinfer properties, such as the classification of a pollen grain ordiatom, in a target image. The rapid and efficient identification ofthis set of images is highly desirable, since image operations, comparedto operations on the database index, are substantially more costly. Ascan be seen in this example, the search methodology is highly effective.

In these examples, the images in the database were processed byextraction of vectors of attribute values, which were further analyzedusing principal component analysis, as described in the section titled“Multivariate Statistical Analysis and Data Clustering”, to determineprojections onto principal component subspaces most conducive toclustering of like images and indexing. The remainder of this subsectionprovides a brief overview of the concepts utilized to construct databaseindices that support similarity-based search and retrieval methods,after which the basics of the statistical analysis and clusteringprocedures utilized in the indexing method are presented.

Views

One aspect of one embodiment of the database architecture is the View,which provides the basis for the implementation of a Search Engine.Referring to FIG. 2, there may be a plurality of Search Engines 210(1)to 210(n). The COTS Database 200 of FIG. 2 can contain an arbitrarycollection of stored objects, which can be arranged in a relationalstructure that, although a factor in the performance of the database,does not have a direct relationship with Views or Search Engines 210.For each View, a specification determines the set of objects in the COTSDatabase 200 that can be accessed using that View, called the ViewableSet. This means that in general not all stored objects may be accessiblefrom a single View. This is reasonable, since some objects can have, forexample, images that are indexed using information derived using a View,while other objects do not.

A View includes a specification for an Attribute Set, which is the setof attributes that can be extracted from any object in the Viewable Set.An attribute value can be any data structure; examples include vectors,sets, and trees of data objects. For example, a “tree of shapes”description and organization of the segments that correspond to aportion of an image can be an attribute value. In a DNA database theattribute value may be a DNA base-pair sequence, an edit sequence, or acollection of alleles associated with loci within an organism's genome.At its most trivial, an attribute value is a number or a symbol. TheSearch Engine 210 that utilizes a View indexes its attribute values, andthe attribute values are stored in the Search Engine's address space.Attribute values are derived from stored objects and can be utilized forrapid comparison of the objects, but note that while two identicalobjects will have identical attribute value sets, identical attributevalue sets do not imply that their corresponding objects are identical.

A View defines an Extractor, which is an algorithm that can be appliedto a stored object within the Viewable Set to produce one or moreattributes, each of which is a value in the Attribute Set. The SearchEngine associated with a View typically applies the Extractor to allstored objects that are in the Viewable Set (as they are stored), andtherefore contains within its address space at least one attribute valuefor each stored object.

A View defines at least one Partition on the Attribute Set. EachPartition defines a Function from the Attribute Set to a finite set ofcategories, or labels, and optionally to a metric space. A metric spaceis a set of values that has an associated distance function d(x,y) thatassigns a non-negative number, the distance, to every pair of values xand y in the metric space. The distance function must satisfy threeproperties: (i) d(x,y)=0 if and only if x=y for all x and y, (ii)d(x,y)=d(y,x) for all x and y, and (iii) d(x,y)+d (y,z)>=d(x,z) for allx, y, and z. If the metric space is defined, the Partition assigns acategory or label to each element of the metric space. Typically, thisassignment is accomplished in a manner that allows an efficientimplementation of an algorithm to compute the category associated withany value in the metric space. The Search Engine 210 utilizes Partitionsto implement a “divide and conquer” search and retrieval strategy,isolating possible matches to a specified request to search to subsetsof categories and implementing a tree-structured index to leaf nodesthat contain attribute values and identifiers of stored objects. Theadvantage of this approach over the capabilities offered by traditionaldatabase technologies is that it supports indexing methods that allowsimilarity-based search and retrieval and depend upon both multivariateand multi-valued (set-valued) quantities; two examples are described inU.S. Pat. Nos. 6,741,983; 7,272,612; and 7,454,411.

The Function typically implements one or more data reduction steps, suchas are described in the section titled “Multivariate StatisticalAnalysis and Data Clustering”. The intent of the data reduction steps isto determine a minimal set of attribute values that enable efficientpartitioning of the stored objects into disjoint collections of roughlyequal size, and, where feasible, to cluster like objects by assigningsimilar attribute values. Therefore, the Function can effect atransformation of the information associated with the stored object intoa useful form that enables at least one of clustering, partition andindexing. As described later, this is typically accomplished through acombination of proper selection of data encoding methods and statisticalanalysis, either using previously acquired and stored data or using adynamic process as new data are acquired and stored.

Properties

Properties are similar to Views but are not utilized to constructindices or Search Engines 210. A Property has specifications of aViewable Set of objects and an Attribute Set of attribute values thatthose objects may possess. Unlike Views, attribute values associatedwith objects are provided by an external source rather than computed byan Extractor. For example, an attribute value can be a manufacturer or ageographic coordinate where the object was found. A typical applicationwould attempt to infer property values for newly acquired objects usinga search for similar objects stored in the database 200 and a model ofhow property values vary or correlate with other attributes of theobject.

Search Engines

Search Engines 210 implement high-performance indices for the database200 of stored objects that allow objects similar to a specified targetto be located and retrieved. Each Search Engine 210 corresponds to atleast one View into the stored data. (An example of a search engine thatutilizes two views is provided in U.S. Pat. No. 6,741,983, where apartition can utilize information from two DNA loci.) Two possibleSearch Engines 210 implement indices of electrical, electromagnetic oracoustic spectra data, and micro-body, for example, charcoal particle ormicrofossil, assemblage data, image data and time series data, i.e.property data of a type for comparison with like previously stored data.A Core Database 200 functionality is capable of supporting more advancedSearch Engines 210. For example, a first Search Engine 210 may bedefined that indices surface sculpturing on pollen grains, allowingreference pollen data to be retrieved that describe grains with similartexture to a target sample. Other Search Engines 210 may be defined toindex the data based upon overall shape, size, and physical attributessuch as apertures. Still other Search Engines 210 may be defined toindex the data on spectral characteristics among acoustic, electrical orelectromagnetic data received, for example, via a passive directionalantenna.

Referring again to FIG. 2, a Client 230 can specify a search contextthat requires similarity in size, shape, apertures, and texture, whichwould be interpreted by the Search Manager 220 to require searches usingmultiple indices (Search Engines) 210 and subsequent analysis andcombination of the results. There may be a plurality of Clients 230. Forexample, a reference to a stored object could be returned only if itwere similar to the target object in at least three of the fourattributes. Another Search Engine 210 could implement an index intospectral data obtained from physical components, retrieving informationabout stored objects of that type that have similar spectra. Informationdescribing both types of objects (and others) can be stored in theunderlying COTS Database 200, whose primary functions are to implementpersistent storage and provide the usual capabilities of a relationaldatabase.

Each Search Engine's index is tree-structured. Operations begin at thetree's root, and paths of descent from each node of the tree areexcluded if no possible matches to the current search specification andtarget can exist on those branches. Leaf nodes of the tree containattribute information and references to objects within the COTS Database200. The attribute data can be used to exclude many referenced objectsas possible matches, leaving a small number of objects that requireadditional analysis—and possibly retrieval from the COTS Database 200—todetermine the final set of matches. In some cases it is possible tomaintain a complete copy of each object within the address space of thesearch engine, if this is required for high performance applications.The Search Engines 210 can support multi-threaded operation, allowingthe simultaneous processing of requests from multiple clients, or from asingle client that has submitted several requests. In one embodiment,write operations, which store new data in the COTS Database 200 ormodify the index structure, block other queries to maintain theintegrity of the index structures. These operations require coordinationacross Search Engines 210, or within the Search Manager 220, because awrite initiated in one index may require modification of data withinanother index that can access the same object(s). An alternateembodiment allows non-blocking writes with subsequent coordination amongprocesses that access overlapping information sets to resolve conflictsor inconsistencies. Referring to FIG. 2, the Search Manager 220 is shownconnected to both Clients 230 and Search Engines 210.

Models

The utility of the similarity database lies in its ability to predictcharacteristics of newly acquired samples using a cumulative database ofpreviously gathered and analyzed materials. It is unlikely that an exactmatch will be found to any particular target, but it is possible tomodel Properties of the new sample using the Properties of similarstored samples. This may be accomplished using interpolation and eitherdeterministic or statistical models, which may be either single- ormulti-variable models, or more complex models may be utilized, asdescribed earlier. The similarity search becomes the first step in thisprocess by restricting consideration of previously stored object data tothose that are most similar to a target object.

A Model includes a specification of Properties, which identifies theViewable Set of stored objects to which the Model can be applied and theAttribute Set that can be utilized by the Model. The model alsospecifies an Algorithm to be used to compute values of a subset of theAttribute Set for a target object, given a set of stored objects and thetarget object. The Model may incorporate an optimization method or anapproximate optimization method to adapt or fit the Model to a subset ofstored objects in the Viewable Set. Note that the attribute values caninclude computed estimates of errors, in addition to the estimates ofvalues such as geographic location, manufacturer, or geographiccharacteristics such as expected nearby land features. Note also thatgeographic location and characteristics could be utilized in aninterface to a Geographic Information System (GIS) such as ARCinfo.

An important feature of a Model is its ability to adapt to newinformation. As additional objects are acquired, analyzed, and stored inthe database, their associate data are available to the Model'sAlgorithm. A search for stored objects and inferred information relevantto a new object is expected to provide more precise answers asadditional data are acquired and stored in the database system. In allcases, the Model should utilize current stored data from objects thatare most similar to a target object's characteristics to developinferences.

Filtering can be used to assess the quality of a model's fit to data(degree with which it accurately describes the relationships among theobjects). For example, one can examine the residuals or innovationsprocesses in filters to determine how accurately the filters model ormatch the behavior of the group of objects. These filtering methods arewell-known in the field of electrical engineering (subfield of systemsand controls), and are also utilized in statistics and businessapplications.

Similarity measures can be used to cluster by similarity and then applymodel(s) to clusters to test hypothetical relationships—with or withouta target object. The target may be a member of the database 200. Forexample, one may perform searches for similar objects contained in thedatabase for all members of the database 200.

SUMMARY

A purpose of the present design is to provide a predictive modelingcapability that is based upon collected reference data. The collectionis dynamic: As new objects are stored in the system, the quality ofinferences improves. The design is not bound to a single modelingparadigm: Models may be as simple as a linear interpolation or a lookupin a database table, but they may be much more sophisticated, usingmultivariate data and optimization, and restricted only by what can becoded in a standard programming language to utilize the structured dataassociated with stored objects. Similarity based search enables theModels to utilize the data that are most similar, using multiplefactors, to a target object, and, since all stored data are available tothe Search Engine 210, the most recent data are utilized, allowing thepredictive modeling capability to remain up to date at all times. Thepatented and patent pending technologies that have been developed at theUniversity of Tennessee allow high-performance similarity-based searchstrategies to be effectively implemented even for very large datacollections, with demonstrated scalability into the hundreds of millionsof stored data objects and demonstrated performance of hundreds to tenthousand completed searches per second utilizing readily availableoff-the-shelf hardware.

Multivariate Statistical Analysis and Data Clustering

Now a method that uses multivariate statistical methods to determineclusters is described that can be utilized to partition portions of adatabase into groups with similar properties and of roughly equal size;see, for example, U.S. Pat. No. 6,741,983. As a result, this methodgenerates partition information that can be incorporated within orassociated with an arbitrary node in a tree-structured database index.The figures are from applying this method to DNA profile data based uponamplification of short tandem repeat (STR) loci.

The raw data associated with objects to be stored (or retrieved) in thedatabase 200 are represented as vectors of numbers. For the DNAdatabase, these numbers are binary and represent the presence (binary“1”) or absence (binary “0”) of a specific STR allele at a locus. Thisencoding scheme is often used for measurements that assign categories,such as “rough”, or “elliptical”, or that represent the presence orabsence of features in raw data, such as signal peaks. Measurement canalso yield floating-point, or real, values, in which case the rawvalues, either scaled or un-scaled, can be utilized. Principal ComponentAnalysis (PCA) of the data is utilized to decrease the dimensionality ofthe raw data by identifying directions of maximum variation in theoriginal data and transforming the data to a new and lower dimensioncoordinate system. For use in a database, coordinates are desired thatresult in discernable and clusterable patterns in the reduced dataspace. Distinct clusters, usually less than 10, can be established usinga clustering method, such as k-means; see, for example, J. T. Tou and R.C. Gonzalez. Pattern Recognition Principles, Addison-Wesley, Reading,Mass. 1992 or k-modes or k-prototypes; see, also, Z. Huang, “Extensionsto the k-means Algorithm for Clustering Large Data Sets with CategoricalValues,” Data Mining and Knowledge Discovery 2, 283-304 (1998). Themembership of each cluster is then identified and recorded. In the DNAapplication, each STR profile belongs to one and only one of theseclusters. Thus, all of the DNA profiles in the database can bepartitioned into these clusters. This partitioning occurs at each levelof the tree-structured database index, enabling a “divide-and-conquer”approach to data retrieval. When searching for data matching a target'scharacteristic, the target can be classified into one of these clustersat each level of the tree. A subsequent search can be restricted tomembers within this cluster. This reduces the search problem byapproximately one order of magnitude at each level of the index tree, asthe search descends the tree.

Principal component analysis (PCA) is a method for analyzing a matrix ofhigh dimension, revealing correlated information and representing itwith a much lower dimensional matrix without sacrificing significantinformation contained in the original data matrix. PCA involves arotation from the original frame of reference to a new frame ofreference, whose axes are given by the principal components from thePCA. The first principal component represents the direction along whichthe variance exhibited by the original data points is maximized and ismade up of a linear combination of the original variables. The secondprincipal component, orthogonal to the first, represents the directionalong which the remaining variance is maximized. Additional principalcomponents are defined in a similar fashion.

To implement PCA, the Singular Value Decomposition (SVD) method can beused to decompose the data matrix, X, into the product of threematrices, in which the columns of the matrix, V, are referred to as the“principal components” of the SVD of the data matrix, X; see, forexample, G. Strang, Linear Algebra and its Applications, 4^(th) ed.,Brooks Cole, Florence, Ky., 2005. Thus,

X=UΣV ^(T)

where U and V are orthogonal matrices, and Σ is a diagonal matrix withnon-negative elements arranged in descending order. The columns of V,being the principal component vectors, represent the coordinates orbasis of the axes of the new frame of reference. The ratio of the squareof each singular value to the total sum of squares of all the singularvalues represents the percentage to the total variation contributed byeach principal component. A Scree plot can be developed to show thecumulative ratio of this measure; an example is shown in FIG. 8.

Since the original data are assumed to be heavily correlated, and thesingular values are arranged in descending order, one can make adecision as to how many principal components to keep in building the PCAmodel to represent the original data. The discarded data along theremaining principal components are regarded as less important and areignored.

Each principal component is of unit length and orthogonal to all otherprincipal components. The principal components are the columns of theright singular matrix, V, of the singular value decomposition (SVD) ofthe data matrix, X, above. Each principal component is expressed as alinear combination of the original variables, with the entries of theprincipal component expressing that particular linear combination. Theabsolute values of all entries are less than or at most equal to 1.Therefore, those entries with relatively large values indicate that thecorresponding original variables exert greater influence along thisprincipal component's direction. The variables with correspondinglyheavy weights are also the ones being correlated in the original dataset.

If the columns of the data matrix, X, are not first mean centered, suchthat the mean of each treated column is zero, then the first principalcomponent reflects the average values of the variables represented inthe new principal component frame of reference. It is then the next fewprincipal components that serve to differentiate between DNA profiles.Therefore, mean centering is an optional step that provides noadditional capability and is not performed here.

After the principal components are found, each data vector can beprojected onto each principal component. The projected vector isreferred to as the scores vector for each sample. The length of thescores vector indicates how closely aligned each sample of that data isto that principal component. The bigger the projection, the better theprincipal component represents the data vector. Thus, data vectors withcomparable projections onto a principal component can be regarded as“similar” to each other, with respect to that principal component. Thosedata vectors with high projected values onto the principal componentindicate that these data vectors are highly aligned with the principalcomponent, therefore representing more of the original variables whichare heavily weighted in that principal component. Similarly, projectionsof data vectors onto each of the succeeding principal components can becarried out to get the scores and their projections onto those principalcomponents.

Because of the different degree of variation exhibited by the datavectors along the different principal components, normalization isnecessary, such that normalized distances from the origin to eachprojected point can be meaningfully compared. The Mahalanobis distancemeasure is employed, in which each projection is divided by thecorresponding singular value. The Mahalanobis distance scores arecalculated as follows:

Mahalanobis Scores=XVΣ ⁻¹ =U

where X represents the original data matrix, and U, Σ and V are from theSVD of X. Postmultiplying X by V performs the projection of the rows ofX (DNA profiles) onto the principal components, with the projectedvectors represented with respect to the principal component axes.Postmultiplying XV by Σ⁻¹ scales each column of XV by the inverses ofthe corresponding singular values contained in Σ. A two dimensional plotcan be used to show the scores onto principal components i and j. Inplotting the scores plot in, say PC2 and PC3, it is the row entries fromthe second and the third columns of the Mahalanobis scores matrix (the Umatrix) that are plotted in a 2-d plot. Henceforth, the Mahalanobisscores shall simply be referred to as the scores. An example of suchplot is shown in FIG. 9, which shows the scores for 10,000 DNA profileswith alleles in the d13s317 and d16s539 short tandem repeat STR locionto the 2^(nd) and 3^(rd) principal components, it is in such a scoresplot that clusterability of the sample points is examined; there werenine clusters in this case.

An aspect is why certain principal component axes, taken for particularportions of the raw data, exhibit good clustering properties, whileothers may not. The answers lie in both the statistical properties ofthe data and the encoding method. For DNA profiles, loci with alleleprobabilities concentrated at just a few alleles (2 to 4) are goodcandidates for generation of distinct clusters. The reason is that withmost of the data having only a few alleles, the joint 2-loci alleledistributions that were used tend to concentrate in those allele pairswith relatively high probabilities of occurrence. Thus, fewer, and moredistinct, clusters tend to be formed. The encoding also plays a role.For example, discrete variables that are numerically encoded tend toenforce a more distinct separation between clusters.

Consider the allele distribution patterns in a large DNA STR data set.If for a specific locus, the probability densities concentrate in only afew, for example 3 out of 10, alleles, then the majority of the DNAprofiles in this data set will have alleles for that locus,corresponding to those with high probability densities. However, some,though in the minority, will still have alleles with low probabilitydensities. Thus, the variance among the DNA profiles associated withthis locus will be higher than those where a large number of alleleshave comparable but low probability densities. The large varianceexhibited by this part of the data will be picked up by the leadingprincipal components of the original data matrix. Recall that theprincipal components of a matrix X are given by the right singularvectors of X, after SVD (the columns of the matrix, V). For a matrixwithout any column mean centering, the first principal componentgenerally gives just the average of the overall data, and therefore isnot useful in differentiating between the points. The second principalcomponent, therefore, is the one that gives the direction along whichthe variance exhibited by the original data matrix is maximum; the thirdprincipal component gives the direction that captures the next maximumvariance, after the component along the first and second principalcomponent have been subtracted off from the original data matrix.

As a result of the above reasoning, the first few leading principalcomponents after the first should exhibit a large contribution fromthose original variables (i.e. the alleles) where the probabilitydensities is large. FIG. 10 and FIG. 11 show the make-up of the secondand the third principal components of the 10,000 DNA profiles at thed13s317 and the d16s539 loci. It is clear from these figures that twoalleles are the dominate factors at each locus in the scores for eachprincipal component. Notice the opposite signs of alleles 11 and 12 ofeach locus in each principal component (PC). If a cluster of the scoresof the DNA profiles projects highly onto the positive direction of PC2,then it means that most members within this cluster have the presence ofallele 12 (the second tall bar of FIG. 10) of d13s317 and allele 11 ofd16s539 (the first tall bar of d16s539 of FIG. 10), and the absence ofallele 11 in the first locus and the absence of allele 12 in the secondlocus, since the signs associated with the latter pair are negative.

The nine distinct clusters can be established analytically by thek-means clustering algorithm, which typically works well for naturallyoccurring data. Other clustering algorithms known in the literature maybe used. Clusters identified by k-means were validated by visualinspection for the DNA database. Memberships within each cluster wereanalyzed to determine the similarity among the members. It was observedthat clusters differ in the combination of alleles at each of the 2 locithat are dominant. From the make-up of the principal components, theprojections of members of each cluster onto each principal component canbe predicted by looking at the pattern of alleles present and absent inthe members of the clusters.

Because the most probable alleles for the d13s317 locus are alleles 11and 12, and the most probable alleles for d16s539 are alleles 11 and 12,the clusters correspond to DNA profiles with various combinations ofpresence or absence of these dominant alleles at these four positions.Boolean expressions can be written that form logical tests on the datato determine cluster assignment. For example, a Boolean expressiontesting for membership in one of the clusters is “(d13s317-allele11) andnot (d13s317-allele12) and not (d13s539-allele11) and(d13s539-allele12)”, where the terms in parentheses are logicalvariables that are true if the corresponding allele is present and falseotherwise. Boolean expressions can be rewritten in various forms andsimplified according to methods that are well known from the fields ofBoolean algebra and logic circuit design.

The Boolean expressions that describe each cluster form a test that canbe applied to any data record. These tests can be utilized to form adecision tree that sequentially applies tests to assign the record to acluster, and therefore to a descent path through the database index,using the methods of inductive inference that were pioneered by J. RossQuinlan; see, for example, “Induction of decision trees,” MachineLearning 1:81-106, 1986. In this case, each node of the database treethat utilizes clusters derived from the multivariate statisticalanalysis method would contain a decision tree specifying the sequence oftests to be applied to DNA profile targets at that node, and thedatabase tree can be rewritten by expanding these nodes andincorporating the decision tree's nodes into the database tree. Agraphical depiction of the database index that results is shown in FIG.12. FIG. 12 is a graphical depiction of a database index dynamicallyconstructed from the results of multivariate statistical analysis,combined with a ranking strategy based upon a measure of similaritybetween objects, to address the needs of various forensic applications.

This method has been utilized for both DNA profile and image indices (asdiscussed later). PCA is utilized to reduce the volume of the raw data,and to focus attention upon a small number of data attributes (principalcomponents) that cause the data to form clusters, resulting in adecomposition of the database. It is possible, however, to utilize toomuch information, in which case clustering will not be achieved. Forexample, from our previous work, the use of PCA methods to analyzeallele information for 16 loci simultaneously does not exhibitclustering. Thus, a key discovery of this work is that it is importantto limit the application of PCA methods to a portion of the availableinformation to achieve good clustering results. A subsequent data fusionstep can be employed to combine information found by searches ofmultiple indices, in a manner similar to the methods utilized incommercial databases to combine the results of searches of two or moretables. In the DNA database used to illustrate the method, theinformation at each level of the database index tree was limited toallele data for two loci.

The factors that determine good clustering and the reason for theclustering have been presented and briefly discussed. Successivepartitioning using different Views (2-loci combinations in this example)at each round very rapidly reduces the number of objects present withineach cluster. Partitioning by PCA clustering can be inserted intosuitably chosen non-terminal nodes of the database index tree, to searchfor matching data objects against a target object. After passing throughthis node, the number of candidate objects that remain to be searched isreduced by approximately one order of magnitude. (Seven to nine clustersusually resulted from PCA clustering of the DNA profile data, in whichthe clusters are about equal in size.)

A very high level of performance is typically achieved using a databaseconstructed in this manner. First, the database's tree-structured indexcan be maintained in memory, as well as vectors of attributes for thestored objects. Second, the operations that must be performed at eachnode of the index are a small number of vector inner products (to obtainthe scores for a search target for each principal component used by thenode), followed by evaluation of a set of Boolean expressions involvinga small number of comparisons. Depending upon the complexity of theapplication, search times for exact matches of microseconds to 10s ofmilliseconds are feasible for a database that resides completely withinthe computer's memory, with longer times required for inexact(similarity-based nearest neighbor) search. The methodology exhibitsgood scalability, with the largest runs to date involving over 100million stored objects. Search times typically scale logarithmicallywith database size. The search time varies with the target and theportion of the database that must be searched (which is determined bythe data associated with the target).

FIG. 13 shows a histogram, using run data from 1999, of search times foran exact search to a specified DNA profile (5019 runs) of a 100,000 DNAprofile database, which an average search time of approximately 2.1microseconds. The methodology can also be parallelized, as described inU.S. Pat. No. 7,454,411, using either a symmetric multiprocessingplatform or a computer cluster.

Analysis of Micro-Particle Assemblages

While rare finds of dinosaur and mastodon bones justifiably attractpublic and scientific interest, knowledge databases related to pastenvironments and the organisms that populated them come from data formuch tinier fossils that are invisible or nearly so withoutmagnification. Such microfossils are preserved in vast numbers in modernand ancient freshwater and marine sediments and soils found worldwide,and are widely used by researchers interested in a variety of themesrelated to the evolution of earth's environments and biota. In themicro-body assemblage database, there may be relational property datafor four microfossils—pollen grains, charcoal particles, diatoms, andforaminifera—that are key indicators of past climate and environmentalchange during the Quaternary and earlier periods of earth's history. Theutility of these microfossils as indicators of past environments stemsfrom 1) their high abundance in nature and 2) the fact that theirdistributions are strongly correlated with geographical andenvironmental factors. These two characteristics make these microfossilsimportant as trace particles in forensic and other efforts aimed atdetermining the source area of objects of interest.

Pollen grains are the structures produced by plants to transfer the malegametes to the female part of a flower. Pollen grains vary in size fromabout 5 μm to more than 200 μm, but most grains are between 20 and 50μm; see, for example, R. O. Kapp, O. K. Davis, and J. E. King, Pollenand Spores, 2^(nd) edition (2000), American Association of StratigraphicPalynologists Foundation. Pollen grains vary in shape, surface texture,and the number and arrangement of apertures through while the pollentube can grow to effect fertilization. These variations follow taxonomy,and make it possible to identify the plants that produced the pollengrains. Most pollen grains can be identified to family or genus, andoccasionally it is possible to identify pollen grains to species.Knowledge of the climate, soil, and other environmental factorsnecessary to support growth of the parent plant provides a basis forinterpreting environmental conditions from the presence of the pollengrain. Although pollen grains of some wind-pollinated plants (such aspines) may be carried long distances by wind, many of these grains fallnear the parent plant. Experimental and other studies reveal that manypollen types are not widely dispersed, such that their presence in anassemblage of pollen grains indicates that the plant was growing nearthe sampling site. Even given the potential long-distance dispersal forthe pollen of some plants, the overall pollen assemblage, expressed asthe percentage representation of different pollen taxa, has beendemonstrated to show a high correlation to the vegetation andenvironmental conditions of the sampling area. This has been shown innumerous studied carried out by palynologists seeking to calibratelong-term pollen records of environmental change by studying the moderndispersal of pollen using pollen traps deployed in different habitats orby collecting pollen present in surface sediments and soils fromdifferent environments; see, for example, P. L. Fall, “Spatial patternsof atmospheric pollen dispersal in the Colorado Rocky Mountains, USA,”Review of Palaeobotany and Palynology 74 (1992) 293-313; C. P. Daviesand P. L. Fall, “Modern pollen precipitation form an elevationaltransect in central Jordan and its relationship to vegetation,” Journalof Biogeography 28 (2001) 1195-1210 and K. H. Orvis, “Modern surfacepollen from three transects across the southern Sonoran desert margin,northwestern Mexico,” Palynology 22 (1998) 197-211. The utility ofpollen grains in forensic applications is well established.

Charcoal particles and fragments ranging in size from that of pollengrains to entire tree trunks and branches are produced by the incompletecombustion of organic matter. Microscopic charcoal fragments are oftentallied along with pollen grains in studies of pollen records ofenvironmental change to provide an index of fire activity; see, forexample, S. P. Horn, “Postglacial vegetation and fire history in theChirripó páramo of Costa Rica,” Quaternary Research 40 (1993) 107-116and M. J. Power et al. (75 total authors including S. Horn), “Changes infire activity since the last glacial maximum: An assessment based onglobal synthesis and assessment of charcoal data,” Climate Dynamics 30(2008) 887-907. Calibration studies have revealed that the abundance ofmicroscopic charcoal particles correlates with both regional and localfire occurrence. Trace charcoal particles can complement forensicstudies of pollen grains, particularly if particular charcoalmorphologies can be discriminated and used to identify the type ofmaterial burned, by comparison to charcoal reference collections; see,for example, K. H. Orvis, C. S. Lane, and S. P. Horn, “Laboratoryproduction of vouchered reference charcoal from small woody samples andnon-woody plant tissues,” Palynology 29 (2005) 1-11. Samples preparedfor analysis of pollen and charcoal may potentially contain distinctivesoot carbon particles or carbon spheres derived from fossil fuel burningand may have forensic potential; see, for example, D. T. L. Alexander,P. A. Crozier, and J. R. Anderson, “Brown carbon spheres in east Asianoutflow and their optical properties,” Science 321 (2008) 833-836.

Diatoms are unicellular algae with siliceous valves with intricateornamentation allowing identification to the species level; see, forexample, R. W. Battarbee, V. J. Jones, R. J. Flower, N. G. Cameron, H.Bennion, L, Carvalho, and S. Juggins, “Diatoms” in TrackingEnvironmental Change Using Lake Sediments, vol. 3, J. P. Smol, H. J. B.Birks, and W. M. Last (eds.), 155-202, Kluwer Academic Publishers,Dordrecht, The Netherlands, 2001. They are similar in size to pollengrains, Diatoms are found throughout the world in almost all aquaticenvironments, both freshwater and marine. Studies of diatoms in surfacesediments of lakes show strong correlations with physico-chemicalfactors such as temperature, pH, nutrient levels (particularly N and P),and salinity. Known relationships between diatom assemblages andenvironmental conditions give diatoms potential utility in forensic andother cases in which materials are to be sourced by environment. Inaddition to diatoms in modern aquatic environments, windblown materialsin drylands of the world may contain diatoms preserved in the sedimentsof ancient dry lakes. If diatoms in ancient sediments are distinct fromthose in modern wetlands, they may provide diagnostic information whenfound on objects of interest.

Foraminifera, or forams for short, are protists which produce tests(shells) that are in most cases composed of calcium carbonate.Characteristics of the tests provide the basis for identification.Forams are primarily found in the marine environment, and are importantmicrofossils in studies of marine sediment cores. They are larger thanthe other microfossils discussed above—some can be up to 3 mm in sizebut most are less than 1 mm. Interpretations from foraminifera are basedon relationships between species distributions and environmentalconditions (for example, benthic or planktonic, and sea surfacetemperatures), and also on studies of oxygen isotope ratios inforaminifera tests. They have been used in forensic studies in comparingmodern marine sediments and rocks; see, for example, Pye, K. Geologicaland Soil Evidence: Forensic Applications. CRC Press, Boca Raton, Fla.,2007.

For all four microfossils of interest, identification is based oncomparison of specimens to reference material and published images anddescriptions. Pollen, microscopic charcoal, and diatoms are examined onglass slides under high-power magnification. Identification by a humanoperator requires examining different focal planes (“focusing up anddown”) to reveal the full, 3D characteristics of the particles. Thisoften requires manipulating the particles to get them to turn over in aliquid mounting medium, to reveal all aspects of their structure andornamentation. This is done by gently tapping the cover slip with atoothpick or other implement. Specimens mounted in a solid medium cannotbe rotated, so only some characteristics may be available foridentification purposes. If multiple examples of the same particle arepresent in the sample, it may be possible to base identifications onfeatures that can be seen in different examples oriented on slides indifferent ways, though care must be taken that examples truly representthe same microfossil. Other processes may hinder identification. Forexample, microfossils may be deformed or damaged, either throughenvironmental wear or during sample collection and processing. Humanexpertise can frequently account for these effects, but they can causeautomated methods to tail. Image segment identification methods that areinvariant to image deformation, or that can operate upon multiple smallsegments of the image that are linked by a segment model may overcomethese difficulties. For example, the tree of shapes image segmentationmodels that we have utilized for preferential segment identification maybe beneficial. We note, however, that a site may also be identified bythe presence of pollen grains from rare plant taxa, or by grains that,even though they are damaged or deformed, have unique and distinctivefeatures. An analogy in the Southern United States is the “dead possum”example—the tail is a unique characteristic by which any Southerner canidentify the remains on the road, even if no other morphologicalcharacteristics used to identify mammals are visible.

The relationships that exist between microfossil assemblages andgeographic and environmental factors provide the basis for using tracemicrofossils on objects of interest to source objects. For best results,databases must be constructed that capture a wide range of details aboutthe sampling site and its surroundings. Only some of this informationmay be present in the original literature report. For all of themicrofossils of interest, the very local conditions of the study sitemay only partially explain the modern assemblage. In the case of pollen,it has been shown that different pollen grains will have differentsource areas, with some types, as mentioned above, potentiallydispersing over larger distances. A single pollen assemblage will beinfluenced by the abundance of plants with poorly dispersed pollen foundclose to the sampling site, and by the abundance of plants with welldispersed pollen found at some distance from the site; see, for example,K. D. Bennett, and K. J. Willis, “Pollen” in Tracking EnvironmentalChange Using Lake Sediments, vol. 3, J. P. Smol, H. J. B. Birks, and W.M. Last (eds.), 5-32. Kluwer Academic Publishers, Dordrecht, TheNetherlands, 2001. Thus, for understanding pollen assemblages it isimportant to know not just the local vegetation cover, but vegetation ofthe surrounding area. For this reason, we expect that it will beadvantageous to construct databases that take into account thevegetation matrix viewed at different scales, for example, thevegetation at the sampling site itself, and within circular areas ofdifferent radii surrounding the study site. Prevailing wind directionsmay suggest that pollen source areas be modeled using ellipsoidal shapesthat are elongated upwind from the sampling site. Prior studies suggestthat whether sites are wooded or open has a strong influence on the sizeof source areas from which they receive pollen input, furthercomplicating the effort that will be required to select, find, and codeenvironmental data appropriate for the interpretation of trace pollenassemblages.

For understanding trace diatom and foraminifera assemblages, importantvariables may be distance to a paleolake, a marine basin, or exposuresof sedimentary rock known to include these microfossils, and may requiredevelopment of a different system for coding data.

Where sufficient data exist in the literature, one may map microfossilassemblages to reveal spatial patterns that can be useful in sourcingobjects of interest, as well as data gaps. “Isopoll” maps similar tothose produced in numerous studies in the eastern United States andrecently for Argentina might be particularly useful though the datadensity required may surpass what can be done without a program ofsample collection, along with data collection for coding environmentalvariables; see, for example, P. J. Bartlein, I. C. Prentice, and T. WebbIII, “Climate response surfaces from pollen data for some eastern NorthAmerican taxa,” Journal of Biogeography 13 (1986) 35-57 and M. M. Paez,F. Schäbitz, and S. Stutz, “Modern pollen-vegetation and isopoll maps insouthern Argentina,” Journal of Biogeography 28 (2001) 997-1021.

The interpretation of source area from trace microfossils may includethe intersection of results based on different aspects of assemblages.Results could potentially be geographical coordinates, but in othercases may be ecological or environmental characterizations, such as,near a wetland area with local wetland plant species and regional if notlocal oak forest, and some biomass burning. Because charcoal fragmentspersist longer in sediments of drylands than pollen grains, they maypotentially provide information on ancient as well as modern humanimpacts, if sediment high in charcoal from prehistoric human activitiesis mobilized by modern land use activities.

Finer scale geographic and environmental sourcing of objects of interestmay include the examination of micro-bodies including microfossils inlarge numbers of environmental samples, as well as objects of interest.Automated methods of micro-particle identification, relying oncontent-based image recognition, may facilitate rapid and extensiveenvironmental and object sampling.

Content-Based Image Recognition (CBIR)

Automated identification of objects such as micro-particles ormicro-bodies using image analysis requires isolation of image segmentscorresponding to each micro-body object and comparison of each segment'sdata against a reference database to identify stored image segments withsimilar properties. This process is illustrated in FIG. 14, where anelectron micrograph of a collection of pollen grains is first segmented(highlighted grain), followed by extraction of the image segment data tosearch a reference database for similar images (middle) tagged withmetadata, to generate a count of grains of each plant (right), which,once all the counts have been computed, can be use to generate theassemblage vector of percentages by plant taxa; the image is publiclyavailable; see http://commons.wikimedia.org/wiki/File:Misc_pollen.jpg.The assemblage vector can then be used to search for similar referenceassemblages in a database, and metadata (properties) associated with thesearch results can be utilized in conjunction with a database Model topredict a geographic location, or characteristics of the location, thatis likely to be associated with the target sample. This subsectionprovides an overview of content-based image search and retrieval, and ofpreferential image segmentation based upon a tree-structureddecomposition and representation of an image called the “tree ofshapes”.

Traditional image search methods are based on keywords. The keywords arechosen in a way that best represents image content, which requiresexpert knowledge and is labor intensive. An automated content-basedimage search capability can be more effective and practical when it isfeasible. Similarity-based search strategies that find images that aresimilar to a target using specified similarity criteria are typical ofcontent-based methods. One approach is to embed data objects derivedfrom the images in spaces such as metric spaces and use the distancefunction or metric as an inverse measure of similarity. Images arerepresented as points in the metric space, and the image indexing andretrieval method may rely upon properties of the triangle inequality ifthe distance function is a metric. Performance is a function of severaldesign decisions, such as the selected image preprocessing algorithms,as well as the index structure and the methods used for data retrieval.The purpose of image preprocessing is to extract a vector of desiredfeatures from the original images. The research efforts at theUniversity of Tennessee have utilized multivariate statistical analysisbased upon PCA to extract feature vectors from images. The featurevectors are embedded in the space, which in this example is a metricspace, and are stored in an index structure that is optimized forsimilarity search. When a search query arrives, similarity searchstrategies based on the triangle inequality are used to retrieve theimages that satisfy the search criterion.

Similarity search based on metric spaces was first introduced inBurkhard, (W. A. Burkhard and R. M. Keller, “Some approaches tobest-match file searching,” Comm. ACM, 16 (4) 1973, 230-236). Thetriangle inequality was first used for similarity search by Koontz, (W.L. G. Koontz, P. M. Narendra, and K. Fukunaga, “A branch and boundclustering algorithm,” IEEE Trans. Comp., C 24, 1975, 908-915).Algorithms based upon this approach can be divided into two categoriesaccording to the way in which they partition the metric space. Somepartition the space using reference points, while others achieve thatbased on Voronoi partitions, (F. Aurenhammer, “Voronoi diagrams: asurvey of a fundamental geometric data structure,” ACM Comp. Surveys(CSUR), 23 (3) 1991, 345-405). This portion of prior research hasfocused on approaches based on reference points. In these approaches,several points in the metric space are chosen, and the distances betweenthese points and all the remaining points are calculated. The metricspace is then partitioned according to these distances. For example,Yianilos implemented vp-tree using this idea; see, for example, P.Yianilos, “Data structures and algorithms for nearest neighbor search ingeneral metric spaces,” Proc. of the 4^(th) Annual ACM-SIAM Symp. OnDiscrete Algorithms, Austin, Tex., 311-321, 1993. In the literature, thenumber of metric computations is typically cited as the criterion ofperformance. However, this is not a good indicator of performance whenpreprocessing steps are utilized and the metric is applied to a featurevector. Image preprocessing is a critical component of similarity searchstrategies that has a significant impact upon overall performance.Search accuracy is also a very important aspect of performance, and mustoften be judged subjectively using human evaluation. The critical issueis whether searches return results that are useful to the end users, andthe choices of metric space and preprocessing steps both influencesubjective search accuracy. New performance criteria that consider bothsearch efficiency and utility have been utilized in our prior researchto guide the development of CBIR databases; see, for example, Z. Shen,Database Similarity Search in Metric Spaces: Limitations andOpportunities. M.S. Thesis, University of Tennessee, August, 2004.

CBIR database design using a metric space approach may be initiated witha choice of preprocessing to extract feature vectors from images, and ofthe metric space. Let X be an arbitrary set. A function d:X×X→

is a metric on X if the following conditions are satisfied for allx,y,zεX:

Positivity: d(x,y)>0 if x≠y, and d(x,x)=0

Symmetry: d(x,y)=d(y,x)

Triangle inequality: d(x,z)≦d(x,y)+d(y,z)

A metric space is a set with a metric, (X,d). Elements of X are calledpoints of the metric space, and d(x,y) is the distance between points xand y.

Image similarity search approaches based on metric spaces embed allimages in a metric space. Similarities between images are evaluatedquantitatively by the metric. Similarity searches are modeled by rangequeries in the metric space, such as: “Find all images within a certainmetric value, or distance, from a specified target.” Given query (q,r)on a data set in a metric space U, where q is the search target and r isthe search range, the goal is to find all objects that are withindistance r from the point q in the metric space, or the set{u_(i)εU|d(q,u_(i))≦r}, which is called the result set of query (q,r).

Search methods based on metric spaces can use tree-structured indexingtechniques to achieve a sub-linear time complexity. At each tree node,indexing divides the data set into several subsets based on similarityrelations between objects. Indexing based on a metric space isequivalent to hierarchically partitioning the space into severalsubsets. Different partition strategies yield different searchperformance. All the existing partition strategies can be divided intotwo categories: methods using reference points, and methods based onVoronoi partitions. The prior work at the University of Tennesseefocused on approaches based on reference points. Partitioning approachesusing reference points choose several reference points in the space andassign one or more of them to each node of an indexing tree. The set ofimages associated with a node is divided into several subsets accordingto the distances between the images and the reference points. Childnodes repeat this process with other reference points until leaves inthe index tree are reached. In this manner, the space of images ishierarchically partitioned into portions of annular regions.

Given the desired tree height h, h reference points {p₁, p₂, . . . ,p_(h)} are chosen. A reference point p_(i) is assigned to the nodes atlevel i of the tree. At level i, the space is partitioned into severalnon-intersecting annular regions R_(ij),j=1,n_(i) centered at thereference point p_(i), defined by a sequence of increasing diameters.Given the set of data points U embedded in the metric space, the annularregions associated with reference point p_(i) are

R _(ij) ={u _(k) εU|d(u _(k) ,p _(i))ε[a _(ij) ,a _(ij+1)]}

where

{a_(ij)}_(j = 1)^(n_(i) + 1)

is an increasing sequence with a_(ij)=0 and a_(inj+1)=∞. The indexingtree can be visualized as h levels of annular regions that overlap. Thefinal partition consists of the intersections of all the annularregions.

FIG. 15 illustrates an example of the partition of a 2-level indexingtree. In (A), the space at tree level 1 is partitioned into threeannular regions R₁₁, R₁₂, and R₁₃ (with a fourth region implicitlyextending from the largest boundary shown in the figure to infinity,which is generally empty). At tree level 2 in (B), the space ispartitioned into two annular regions R₂₁ and R₂₂. The final partition(C) of the 2-level indexing tree is produced by the intersections ofthese five annular regions. There are eight subsets in the finalpartition (C) (not including the implicit regions that extend toinfinity).

Image similarity search methods that use indices based upon referencepoints may use the triangle inequality to rule out partitions, andtherefore paths of descent in the index tree, that can not contain asolution. The search request propagates through the tree-structuredindex, and a candidate set is generated. A result set, which is a subsetof the candidate set, is obtained by exhaustively searching thecandidate set. The candidate set of query (q,r) is found using thetriangle inequality. In FIG. 16, three points, a reference point p_(j),the query target q, and an object u_(i) are located in the metric space,demonstrating the triangle inequality in similarity search. The triangleinequality relates the values of the metrics, or distances, asrepresented in the figure by lines, by the inequalities:

d(q,u _(i))≦d(u _(i) ,p _(j))+d(q,p _(j))

and

d(q,p _(j))≦d(u _(i) ,p _(j))±d(q,u _(i))

d(q,p _(j))−d(u _(i) ,p _(j))≦d(q,u _(i)),

or

d(q,p _(j))−d(u _(i) ,p _(j))≦d(q,u _(i))≦d(q,p _(j))+d(u _(i) ,p _(j)).

If u_(i) belongs to the result set, it should satisfy the searchcriterion

d(q,u _(i))≦r,

or

d(q,p _(j))−r≦d(u _(i) ,p _(j))≦d(q,p _(j))+r.

Therefore, a necessary condition SC that must hold in order for thesearch criterion to be satisfied by u_(i) is,

${SC} = {\overset{k}{\bigcap\limits_{j = 1}}\left\{ {u_{i} \in U} \middle| {{d\left( {u_{i},p_{j}} \right)} \in \left\lbrack {{{d\left( {p,p_{j}} \right)} - r},{{d\left( {q,p_{j}} \right)} + r}} \right\rbrack} \right\}}$

The candidate set Cand is the union of all the stored objects lyingwithin partitions that intersect the search criterion SC,

${Cand} = {\underset{i = 1}{\bigcup\limits^{i}}\left\{ P_{i} \middle| {{P_{i}\bigcap{Sc}} \neq \varnothing} \right\}}$

where t is the total number of partitions. Once the search request hasbeen restricted to the candidate set, the candidate set is scannedexhaustively to get the result set,

Res={u _(i) εU|u _(i)εCand

d(u _(i) ,q)≦r}

FIG. 17 illustrates an example of processing a search query (q,r) on atwo level index tree based upon reference points P₁, P₂, P₃. In (A),three subsets intersect with the search criterion, and in (B) twosubsets intersect with the search criterion. The shaded area in (C),which is the intersection of the two shaded areas in (A) and (B),represents the candidate set.

One component of the search time is typically proportional to the sizeof the candidate set, due to linear search. A second component is due totraversal of the tree, and is typically logarithmic in the size of thedatabase, and a third component is due to computation of the metricdistance from the query to each reference point. This is summarized bythe equation

T=N _(ref×T) _(metric) +N _(cond) ×T _(metric) +T _(tree)=(N _(ref) +N_(cand))×T _(metric) +T _(tree)

where N_(ref) is the number of reference points, N_(cand) is the numberof objects in the candidate set, and T_(tree) is the tree traversaltime. Let N_(metric)=N_(ref)+N_(cand), which is the total number ofmetric evaluations. Since metric computations are usually more timeconsuming than the time required to traverse the index tree, T_(tree)can be neglected. In most situations, N_(cand)>N_(ref) by a wide margin,so the size of candidate set is the dominant component and the searchtime is primarily determined by N_(cand).

The design of a CBIR database is typically an iterative process, withtrade-off studies performed on a sample of representative images todetermine the optimal preprocessing strategy and embedding in a metricspace. This process needs to be guided by quantitative evaluations ofthe performance of candidate designs. Usually, the number of metriccomputations determined by the candidate set size is used as thecriterion to evaluate search performance. However, this criterion onlyworks for comparing different search methods that produce the sameresult set. In other words, the comparison of N_(metric) is feasiblewhen the search results are the same. Different image preprocessingmethods, index structures and retrieval strategies will yield differentresult sets. Therefore, a new criterion that considers both thecandidate set size and result set size is required. The ratio betweenN_(res), the number of results of a search, and N_(cand) has been chosento meet this requirement. A high quality search strategy should yield alarge value for the ratio N_(res)/N_(cand). In other words, N_(res)should be close to N_(cand), which means few unnecessary metriccomputations are performed during the search. The value ofN_(res)/N_(cand) also measures the efficiency of a search strategy. Inorder to compare the performance across different data sets, normalizedsearch ranges are used. A normalized search range is the ratio betweenthe search range and the average distance between all the storedobjects, or r/μ, where the average distance μ is

$\mu = \frac{\sum\limits_{i = 1}^{N_{total}}{\sum\limits_{j = {i + 1}}^{N_{total}}{d\left( {u_{i},u_{j}} \right)}}}{N_{total} \times {\left( {N_{total} - 1} \right)/2}}$

where N_(total) is the total number of objects stored in the database. Afigure that illustrates the values of N_(res)/N_(cand) against differentr_(normalized) is used to evaluate the performance of different metricsand data extraction methods. In such a figure, the area under the curveof N_(res)/N_(cand) indicates the performance, and a larger area means abetter performance with respect to search efficiency. FIG. 18 is anexample figure comparing performance of two different data extractionmethods a and b.

The area under curve a is larger than that under curve b. Thus, thesearch performance of using data extraction method a is better than thatusing b. In order to make this criterion more suitable for practicalapplications, an improved performance evaluation method is provided.Assume the search ranges are distributed exponentially, i.e.,

p(r _(normalized))=γe ^(γ) ^(normalized)

for a positive constant γ. The search performance for search rangessmaller than r_(max) can be evaluated by a weighted integration,

${\varphi \left( r_{{ma}\; x} \right)} = {\int_{0}^{r_{{ma}\; x}}{\frac{N_{res}\left( \hat{r} \right)}{N_{cand}\left( \hat{r} \right)}\gamma \; ^{{- \gamma}\; \hat{r}}{\hat{r}}}}$

The performance characteristic measured by φ(r_(max)) is expected searchefficiency over exponentially distributed search ranges less thanr_(max). The value of r_(max) is assumed to be sufficiently large thatthe contribution by the tail of the distribution can be neglected.

The numeric value of φ(r_(max)) provides a method of comparing searchefficiency across candidate database designs. Another critical measureof performance, which tends to be highly subjective, is the utility ofsearch results. In other words, does the search method return resultsthat are useful to users? A method that we have utilized in a priordesign for a database of 1,092 photos of rowing teams is used as anillustration. The photos in this data set belong to various types,including photos looking down on a boat with rowers, photos of boats ina race at different locations and events, taken from the side, andphotos of a rowing team in groups. For the evaluation of databaseutility, all the photos were divided into 8 types, and 3 photos werechosen from each type. These selected photos were used as searchtargets, defining 24 searches. Four example photos belonging todifferent types are presented in FIG. 5. The search results were judgedmanually by one of the investigators, and a score was assigned to thequality of the returned results for each search. The scores for searchefficiency and search utility were utilized to compare among a selectionof image preprocessing steps utilizing PCA methods to extract featurevectors, and different metric space embeddings, enabling theimplementation of a database that provided both acceptable utility andhigh performance.

A CBIR database may be used to store raw images, but it is likely to bemore effective in the identification of micro-bodies or micro-particlesif the images are first segmented. An ideal segmentation would createimages containing, for example, individual pollen grains or diatoms withno background or obscuring data. This can be done manually, but partialor total automation of the image segmentation step may use apreferential image segmentation algorithm based upon “tree of shapes”descriptions of the image and image segments, as described in detail inY. Pan, Image Segmentation using PDE, Variational, Morphological andProbabilistic Methods, PhD Dissertation, Electrical Engineering,University of Tennessee, December, 2007, incorporated by reference inits entirety. This representation provides a hierarchical tree for theobjects contained in the level sets of the image. The hierarchicalstructure is utilized to select the candidate objects from the image.The boundaries of the selected objects are then compared with those ofobjects selected from prior images. By means of the tree of shapes andcurve matching, the proposed method is able to preferentially segmentobjects with closed boundaries from complicated images. It is morestraightforward to utilize prior information in this way than with curveevolution methods, and there is no initialization problem. Furthermore,the method is invariant to contrast change and translation, rotation andscale. The method has been shown to work in the presence of noise.

The preferential image segmentation algorithm is illustrated by example.An intuitive description of the algorithm is to construct the trees ofshapes for both a target and a candidate image that are to be compared.The candidate image would correspond to a reference image of a pollengrain in a database, while the target image would correspond to a sampleto be analyzed. Both images are segmented into a tree of shapesdescription, which is a nested collection of upper (or lower) levelsets; see, for example, L. Ambrosio, V. Caselles, S. Masnou, and J. M.Morel, “Connected components of sets of finite perimeter andapplications to image processing,” Journal of the European MathematicalSociety, 3(1):213-266, 2001. The objective is to find a node within thetree of shapes description of the candidate image that is the root of asub-tree that matches the tree representation of the target (reference)image to within a specified accuracy.

FIG. 19 shows two examples of preferential segmentation which are shownhere to illustrate the performance of the method; see Pan for a moreextensive evaluation where (A) represents the target image of a trucksegmented in a candidate video frame (C) and (B) is a target image of anelectronic component segmented in image (D). Image (C) is a surveillancevideo frame that was segmented to find a match to target image (A). Theonly matching segment that was returned was the segment containing theimage of the truck (A), outlined in red (heavy line). Image (D) is aphotograph of a collection of electronic parts. Image (B) is aphotograph of one of these parts taken with a different orientation. Thepreferential image segmentation algorithm segmented the correct part,outlined in red (third component from left bottom) in image (D),demonstrating the algorithm's tolerance of rotation and scaling.

In one embodiment, preferential image segmentation may be utilized toisolate images of individual micro-bodies for identification. Featurevectors are extracted from each isolated image and utilized to query adatabase of reference images and associated metadata in order to selectthe most similar reference data to each particle from the database andidentify the charcoal particle, pollen grain, diatom, foraminifera, orother particle. Micro-body assemblage data can be constructed for eachsample from these identifications, substantially reducing the humanlabor necessary to process samples.

Images obtained using optical microscopy have an extremely shallow depthof field relative to the diameter of, for example, a pollen grain.Microfossils have three dimensional structure, and any projection ofthis structure into a two dimensional image represents a loss ofinformation. The limitations of the shallow depth of field can bereduced using a sequence of images as the focal plane is moved, and bothconfocal and scanning electron microscopy SEM provide greater depth offield, with SEM microscopy having a significant advantage over standardoptical techniques when surface texture is used as a discriminatingfeature. A 3D representation of a microfossil can be constructed frommultiple 2D images taking at various orientations, but requiresadditional reference data. Past automated identification efforts haverelied upon image analysis algorithms that are specific to shape ortextural features, or artificial neural networks (ANN). Of the ANNapproaches, France et al. (I. France, A. W. G. Duller and G. A. T.Duller, Software Aspects of Automated Recognition of Particles; theExample of Pollen, Image Analysis, Sediments and Paleoenvironments, P.Frances (ed.), Kluwer (2004) 253-272) appear the most promising. Franceet al. utilize a 3-layer network, using Gabor filters to detect edges,followed by a layer to piece edges together into an object and a finallayer for identification. During training, their approach adds newobjects that cannot be classified to the set of classes, allowing thealgorithm to adapt to newly presented data (if done in training).

CBIR using similarity search is applied in one embodiment formicro-particle recognition, allowing Model-based prediction of aparticle's taxon using the most similar reference data available. Thisapproach provides a natural growth path as new data are added to thereference collection, obviating the need for new algorithms orretraining of classifiers. The objective is classification of eachmicro-particle and subsequent calculation of one or more micro-bodyassemblages for each sample, using a system and methodology that cangrow with the reference collection, producing better predictions withgreater accuracy over time. Once a micro-particle assemblage has beenobtained from a sample, the micro-body assemblage database (MAD)constructed of data for the micro-particle assemblages can be queried todetermine similar micro-body assemblages within the reference database.These reference data can be utilized in conjunction with models topredict geographic location or other characteristics of the object or ofthe local environment.

Measured properties of spectral/acoustic data, micro-body assemblagesand images of objects can be utilized, in conjunction with respectivedatabases that support search and retrieval based upon similaritiesamong objects, to provide information about geographic location andother properties of a sampled target object. ESD and MAD are examples ofdatabases that can build upon existing technologies that have beendeveloped to implement high-performance similarity search engines.

Thus, there is provided an automated analysis and identification ofmicro-particles, larger objects and the like which may be found in tracequantities of or even be the target object. Content-based imageretrieval (CBIR), and associated databases which may containphotographs, X-rays, MRI image data, infrared image data and the like,is a relatively new technology that has undergone rapid evolution overthe past decade. The literature on automated microfossil identificationfocuses primarily on two approaches: structural/statistical techniques,and methods based upon neural network classifiers. The first approachrequires image analysis algorithms tailored to resolve specificfeatures, while neural network methods have difficulty extending as thereference data expand, without lengthy retraining. CBIR, combined withpreferential image segmentation, will be effective in reducing theburden placed upon the classification step by focusing the classifier'sscope to the set of reference data and features (for example, aperturesand sculpturing on pollen grains) most similar to a target sample'simage(s) and most likely to lead to accurate identification.

Referring to FIG. 20, there is shown a data modeler platform inaccordance with one embodiment. In accordance with FIG. 20, client 2030,server 2000 and storage 2010 can be combined as a single unit (e.g., acomputer or laptop), or separate units (multiple computers thatcommunicate using, for example, a network). Each unit is able tocommunicate with either a user (using, for example, a keyboard, mouse,and display, not shown) or a computer or device (using, for example, awired network 2020 such as Ethernet or a wireless communicationsinfrastructure such as IEEE 802.11 or a packet data network 2020 such as3G cellular or PCS), which can optionally provide an interface to auser.

The server 2000 may be implemented using several networked servers withdifferent functions allocated to each server. For example, a server 2000might be utilized for each database index. A separate server, ormultiple servers, not shown, might also be utilized to processtransactions and communications with clients 2030(1) and 2030(2). One ormore servers 2000 might be utilized to control specialized data or imageacquisition equipment such as microscopes, cameras, and scanners.Alternatively, some or all of these servers might be implemented asvirtual servers in one or more physical servers using software such asXen (http://www.xen.org/), VMware ESXi (http://www.vmware.com/), or SunxVM Ops Center(http://www.sun.com/software/products/xvmopscenter/index.jsp).

As another alternative, the server 2000 could utilize a computer withmultiple processors and/or multiple cores having either a symmetricmulti-processing (SMP) or non-uniform memory access (NUMA) architecture.Storage 2010 can be contained within the server, or separate, as wouldbe the case, for example, when a network-attached storage (NAS) deviceor storage appliance was used. Redundant storage systems may beutilized; example technologies include RAID and Sun ZFS, and may includeredundant hardware, power, and network pathways. The server 2000 may, byway of example, be a Sun Fire X2200 M2×64 Server containing twoquad-core AMD model 2376 processors, 32 GB of memory, two 146 GB SAShard disk drives, and a DVD-ROM. The bus system 2005 may include a SunStorageTek™ 8-port external SAS PCI-Express Host Bus Adapter that ishoused with the server 2000 as an interface to an external storage array2010. The external storage array 2010 may be a Sun Storage J4200 arraywith 6 TB of storage. The work station systems include, for example, sixSun Ultra 24 Workstations with 22″ LCD monitors, which can be used asclients 2030 to the server 2000. Racking for the system may include anequipment rack with a power distribution unit and an uninterruptiblepower supply. A network switch for network 2020 is not shown but may beimplied from their common utility in, for example, a local area network,a wide area local network or any telecommunications network known in theart. A typical network switch for the system of FIG. 20 may be theNetgear JGS524 Prosafe 24-Port Gigabit Ethernet Switch, with compatible(CAT-5e or CAT-6) cabling. If one were to use network attached storage(NAS) such as iSCSI or a network storage device such as the Sun 7200Unified Storage System, a second network switch might be utilized toseparate data traffic between the storage system 2010 and the server2000 from data traffic between the server 2000 and other computers orclients 2030.

System components will now be discussed with reference to FIG. 21.Referring to FIG. 21, the system supporting databases and prediction ofproperties of objects has at least one processor 2100, but may have morethan one processor, and the processor may implement more than oneprocessor core. The processor has access to memory 2110, which is usedto store index structures that enable rapid access to stored objectsthat have similarities to the attributes of a target object specified ina query. Storage 2120 is utilized to provide persistent memory and toserve as a repository for information that does not need to be accessedas efficiently (rapidly) as the in-memory objects. For example, imagesmay reside in storage 2120 while descriptions of the shapes of segmentsof these images or other attributes of the images may reside in memory2110. One or more clients 2140 can submit queries to the server'ssoftware, which are interpreted by the processor 2100 in order toperform searches using the index structures that are resident in memory2110 and, possibly, the data contained in the storage 2120. Results arereturned by the processor 2100 to the clients 2140 via network 2130.Users can interact with the system through the client(s) 2140 usinginput devices such as a keyboard 2142 and mouse 2144 and output devicessuch as a display 2146. All of the components may be implemented in asingle computer system such as a laptop, desktop, or server, or they maybe implemented in separate computers that interact using acommunications medium such as a wired or wireless network 2130.

A data acquisition device 2150 may be connected to either a client 2140or a server 2100, 2110, 2120 using an interface such as a serialinterface, Ethernet, a data acquisition and control card, a universalserial bus (USB), or a FireWire bus or network 2130. Example dataacquisition devices include scanners, microscopes (optical, electron, orconfocal), cameras (still image or video), antennas, infrared sensors,acoustic sensors, laser rangefinders or scanners, passive microwavesensors and spectroscopic instrumentation or related field portabledevices. The interface 2130 to the data acquisition device 2150 may bebi-directional, meaning that the server or client can control theoperation of the data acquisition device 2150 to, for example, locateand examine portions of a specimen that is subject to analysis. The dataacquisition device 2150 may utilize a wireless, wired, acoustic, oroptical communications link to control a remote device and/or acquireinformation from a remote device.

All United States and foreign patents and articles whose citations areprovided above should be deemed to be incorporated by reference as totheir entire contents for the purposes of understanding the underlyingtechnology behind an embodiment of a method and apparatus for predictingobject properties using similarity-based information retrieval andmodeling. The embodiments of a method and apparatus for predictingproperties using similarity-based information retrieval and modelingdescribed above should only be deemed to be limited by the scope of theclaims which follow.

What we claim is:
 1. A method of detecting a pattern of one of afraudulent activity and a criminal activity by recording an object in atransaction database, the object comprising at least one suspicioustransaction in a transaction database, the transaction database beingcoupled to processor search manager apparatus, the processor searchmanager apparatus comprising a processor and memory, the memory coupledto the transaction database, the method comprising testing by saidprocessor of hypotheses representing a possible organizational structureof a criminal or fraudulent conspiracy, said transaction databasecomprising one of a similarity-based and a nearest neighbor search; andapplying at least one model to results returned from the database inresponse to a query, the query requesting objects in said transactiondatabase most similar to a target object, the queried objects,responsive to the query, being most similar to each other, the testingincluding testing at least one hypothesis and detecting the pattern ofone of the fraudulent activity and the criminal activity; the patterndetection comprising a prediction of attributes including two of gender,marital status, occupation and age of an offender of the suspicioustransaction and member of the criminal or fraudulent conspiracy.
 2. Themethod of claim 1, the transaction database comprising purchasingpatterns of a plurality of consumers, the method further comprisingapplying the model to consumer members of a cluster of similar consumerbehavior to obtain a relationship comprising a flow of one of capital,goods and services between consumer members of the cluster.
 3. Themethod of claim 1 an organizational structure hypothesis comprisinginteractions among a plurality of communication devices forming nodes ofa communications network.
 4. The method of claim 1 wherein thetransaction comprises a medical record of medical data of a medicalclaim of a consumer of medical services to detect a pattern offraudulent activity of the consumer.
 5. The method of claim 1, thetransaction database comprising a financial transaction database, themethod comprising predicting an instance of medical insurance fraud byone of a consumer and an enterprise responsive to a query of thefinancial transaction database.
 6. The method of claim 1 wherein thepattern comprises equities that move as a group in a first transactionor move as a plurality of transactions delayed in time with respect toone another.
 7. The method of claim 1 further comprising miningtransaction information stored in the transaction database to identify acluster representing a similar behavior including a purchasing patternof one of a consumer and an enterprise.
 8. The method of claim 1 furthercomprising a plurality of databases, the plurality of databasesincluding the transaction database, a spectral property database, amicro-body assemblage database and an image database, the method furthercomprising predicting one of a source, an origin and a location of oneof an object responsive to the query with reference to the spectraldatabase, the micro-body assemblage database or the image database.
 9. Amethod of detecting a pattern of a fraudulent activity by recording anobject in a transaction database, the object comprising at least onesuspicious transaction in a transaction database, the transactiondatabase being coupled to processor search manager apparatus, theprocessor search manager apparatus comprising a processor and memory,the memory coupled to the transaction database, the method comprisingtesting by said processor of hypotheses representing possibleorganizational structures of fraudulent activity, said transactiondatabase comprising one of a similarity-based and a nearest neighborsearch; and applying at least one model to results returned from thedatabase in response to a query, the query requesting objects in saidtransaction database most similar to a target object, the queriedobjects, responsive to the query, being most similar to each other, thetesting including testing at least one hypothesis and detecting thepattern of fraudulent activity.
 10. The method of claim 9, thetransaction database comprising purchasing patterns of a plurality ofconsumers, the method further comprising applying the model to consumermembers of a cluster of similar consumer behavior to obtain arelationship comprising a flow of one of capital, goods and servicesbetween consumer members of the cluster.
 11. The method of claim 9 anorganizational structure hypothesis comprising interactions among aplurality of communication devices forming nodes of a communicationsnetwork.
 12. The method of claim 9 wherein the transaction comprises amedical record of medical data of a medical claim of a consumer ofmedical services to detect a pattern of fraudulent activity of one ofthe consumer and an enterprise.
 13. The method of claim 9, thetransaction database comprising a medical transaction database, themethod comprising predicting an instance of medical insurance fraud byone of a consumer and an enterprise responsive to a query of thetransaction database.
 14. The method of claim 9 wherein the patterncomprises one of equities and money that move as a group in a firsttransaction or the one of equities and money move as a plurality oftransactions delayed in time with respect to one another and thepossible organizational structure.
 15. A method of detecting a patternof a criminal activity by recording an object in a criminal transactiondatabase, the object comprising at least one suspicious criminaltransaction in the criminal transaction database, the criminaltransaction database being coupled to processor search managerapparatus, the processor search manager apparatus comprising a processorand memory, the memory coupled to the criminal transaction database, themethod comprising testing by said processor of hypotheses representing apossible organizational structure of a criminal conspiracy, saidcriminal transaction database comprising one of a similarity-based and anearest neighbor search; and applying at least one model to resultsreturned from the database in response to a query, the query requestingobjects in said criminal transaction database most similar to a targetobject, the queried objects, responsive to the query, being most similarto each other, the testing including testing at least one hypothesis anddetecting the pattern of the criminal activity; the pattern detectioncomprising a prediction of attributes including two of gender, maritalstatus, occupation and age of an offender of the criminal transactionand member of the criminal conspiracy.
 16. The method of claim 15, themodel comprising one of a Bayesian model and a belief network, the modelcomprising a graph of hypothetical relationships between objectsresponsive to the query.
 17. The method of claim 30 further comprisingperforming a search of the transaction database against itself todetermine a list of most similar clusters.
 18. The method of claim 15further comprising predicting one of an individual offender and anentity engaged in an illicit enterprise.
 19. The method of claim 15further comprising predicting one of a type of material used toaccelerate a fire and a type of carried weapon.
 20. The method of claim15, the hypotheses comprising first and second hypothesis comprisingfirst and second likelihoods, the set of first and second likelihoodscomprising a likelihood ratio.